A Multimethod Evaluation of Simulated Conversations as an Instrument for Assessing Pre-Service Teachers’ Communication Competence

• In Event: 63.065 - Performance-Based Strategies to Train and Assess Teacher–Parent Conversation Competencies

Abstract

1) Objectives

In this study we evaluated the validity of “simulated conversations” to diagnose pre-service teachers’ communication competence in teacher-parent conversations. For this purpose, we created school-related scenarios in which participants conducted simulated conversations with trained actors portraying parents. Focusing on two aspects of construct validity, we addressed the research question whether the communication competence demonstrated in simulated conversations (as rated by trained external observers) is positively correlated to a) ratings by the simulated parents and self-assessments of the pre-service teachers, and to b) external variables such as prior knowledge.

2) Theoretical Framework

Communication competence in teacher-parent conversations can be considered a hierarchical construct comprising teachers’ abilities to structure a conversation transparently, drive it towards a solution, and to establish a relationship with the parents (Gartmeier et al., 2011). There is convincing evidence that teacher-parent cooperation has a positive impact on students’ school success and social development (e.g. Harvard Family Research Project, 2007). However, teacher education frequently fails in preparing teachers for teacher-parent communication (Flanigan, 2005). Fostering such competences requires training programs as well as diagnostic instruments that allow, firstly, to validate these training programs and, secondly, to assess (pre-service) teachers’ level of communication competence and identifying aspects that require improvement. One promising diagnostic method is simulated conversations with trained actors in predesigned scenarios, an established approach in medical education. There is evidence from medicine that simulated conversations deliver reliable scores and have a high prognostic validity (Association of Standardized Patient Educators, 2014). Thus, we transferred the idea of simulated conversations to the teaching domain.

3) Methods and Data Sources

N=96 pre-service teachers conducted two simulated conversations each. Conversations were videotaped. Two trained, independent raters rated these videos according to the stated competence aspects (structuring, problem solving, relationship-building) using high-inference ratings (9 items, 5-point Likert-scale). Confirmatory factor analysis showed that the expected hierarchical model with the three competence facets as first order factors and a second order factor ‘general communicative competence’ fits the data adequately [χ2(24)=43.54, p<.009, RMSEA=.071, CFI=.971, SRMR=.057]. All factors showed satisfactory composite reliability (.69

4) Results and Conclusion

(a) As expected, the external observers’ ratings were positively correlated to the simulated parents’ ratings (r=.46, p<.01) and to participants’ self-assessments (r =.21, p=.04). The latter effect size is in the expectable range for correlations between external observations and self-report measures (Eid et al., 2010). (b) The trained observers’ ratings showed the expected positive correlations to the external variables prior knowledge, (r=.34, p<.01) , and final school degree (r=.39, p<.01).
These results are promising and provide first evidence for the construct validity of simulated conversations for diagnosing the communication competence of pre-service teachers in teacher-parent conversations.

5) Scholarly Significance

The results are of theoretical and practical relevance for performance-based assessments in teacher education. In comparison to traditional assessment methods like questionnaires, simulated conversations produce observable performance in standardized but authentic task situations (Shavelson, 2013). This allows a more accurate assessment and consequently enables teacher educators to provide differentiated formative feedback to students to prepare them better for communication with parents (Dotger et al., 2009).

Authors

• Anne Birte Wiesbeck, Technische Universität München

• Johannes Bauer, Technische Universität München

• Martin Gartmeier, Technische Universität München, TUM School of Educaiton

• Manfred Prenzel, Technische Universität München