Creating Context and Ensuring Authenticity in Scenarios for Measuring Domain-Specific Problem-Solving Competences

• In Event: 65.078 - "Context Matters!" Antecedences of Vocational Education and Training Competencies

Abstract

In our study we focus on the necessity of creating context for test situations and implementing this authentic context when testing prospective industrial clerks within the German system of vocational education and training (VET). The study is funded by the German Government.
Following Chomsky (1965), competence is seen as a latent construct that can be measured by observing and interpreting manifest behavior in situ. Hence, in order to measure competence one should create test situations that are as consistent as possible with the challenges in real work situations. Moreover, such test situations should allow the observation of all hypothesized facets of the underlying competence construct (Sembill, Rausch & Kögler, 2013).
In line with various authors we perceive competence as a multifaceted construct of individual dispositions (Sembill et al., 2013, p. 200; Breuer, 2006; Weinert, 2001): Accordingly, motivational and emotional facets are crucial for action regulation and thus guide performance. A sufficient motivational basis, for instance, results in higher goal commitment and prevents from being distracted or giving up early. Furthermore, interest and self-concept influence problem-solving performance in the workplace. Thus, tolerating uncertainty, maintaining positive emotional states, and managing one’s attentional resources are facets of our approach to measure problem-solving competence.
We explore the influence of “authentic test situations” and “motivational and emotional states” on “problem-solving strategies” as well as on “domain-specific problem-solving competence” at commercial workplaces. Therefore, we developed a computer-based test environment that resembles typical office software (file system, email-client, spreadsheet-program, notebook, calculator). Our (N=123) test takers overtake the role of a young employee of a bicycle manufacturer. The company does actually exist and supported us by providing authentic materials and actual problem cases. After an introduction into the test environment, three complex problems are presented. These 30-minute problem scenarios cover typical business tasks. Each problem scenario starts with an email of a superior. Additionally, plenty of relevant but also irrelevant information are provided e.g., such that does not help solving the professional problem at hand but exerts some distracting potential - as in daily work situations
Data derive from several sources and multiple instruments: The test environment logs every mouse click of a test taker. After each problem scenario, the participants rate the authenticity and interestingness of the scenario. Furthermore, they are asked about their invested effort compared to a real-life task at the workplace. Additional data derive from ‘embedded experience sampling’ (EES), which aims at collecting data on motivational and emotional facets by simulating authentic social interactions in the workplace, for instance, a friendly colleague asking how things are going.
By correlation- and regression-analyses we explore the relationship between situational contexts, individual prerequisites and implied problem-solving strategies as well as the overall problem-solving competence (based on Item Response Theory). The results will be discussed with regard to the necessity of context in test tasks. First steps of our pre-analyses show, that context matters. Our results will enhance the scientific discussion on modelling and measuring competence. They will also give practical hints for fostering domain-specific competences in VET.

Authors

• Kristina Koegler, University Frankfurt am Main

• Andreas Rausch, University of Bamberg

• Eveline Wuttke, Johann Wolfgang Goethe University

• Juergen Seifried, University of Mannheim

• Julia Warwas