Effect of Online Longitudinal Progress Test Feedback

• In Event: 60.044 - The Best Predictor of the Future May Be the Past, but How Accurate Are the Predictions? Estimating Professional Competence Using Diverse Methodologies

Abstract

Perspective: Progress tests offer the potential for rich feedback to students and teachers. A progress test is a cross sectional test on the complete domain that is administered to students of all levels at regular intervals of time, hence offering a longitudinal view on knowledge development. Profile scores may be provided for all subs-cores, derived from the progress test blueprint and individual performance may be benchmarked against peer group performance. This may allow learners to explore their strengths and weaknesses. In 2009 we introduced a web-based dashboard called ProF that allows students to view their progress test results online. The system shows the longitudinal development and details per tests in several formats. It displays the results of a student against the aggregated development of their peer group (both locally and on a national level). ProF provides profile graphs and offers the possibility to drill down into sub-scores. ProF was introduced to provide a flexible and personalized access to the longitudinal feedback information and was intended as an improvement over the paper-based feedback as provided until then.

Objective: In the current research we aimed to investigate whether the ProF system helps students in processing their progress test results and improves their results.

Methods and Data Sources: Five Dutch medical schools cooperatively create and administer a progress test four times each year. The test is taken on paper by all students from year 1 to 6 at the same time (except for one school). Each test consists of 200 MCQ questions that cover the complete medical knowledge required for a graduate student. Students are allowed to answer ‘don’t know’. The total score of a progress test is based on formula scoring (roughly correct minus incorrect score) and is expressed in percentage. Regression analyses were used to examine use of the system and progress test scores, controlling for student level.

Results: The average knowledge growth in this period was 9.9 percentage points. Regression analysis showed a significant coefficient of 1.06 for ProF usage in study year 3, which means an additional growth of 5.3 percentage point with 5 uses. There is no significant difference between schools, but for study year 5 and 6, the effect is significantly lower.

Significance: Results support our hypothesis that ProF helps students analyze progress test feedback and improve their scores. Follow-up studies are needed to investigate further the causes of the observed effect. The study shows that providing a dashboard for longitudinal feedback can help students to identify their strong and weak points, to change their study strategy, and to improve their performance, all important aspects of an effective learning environment.

Authors

• Jeroen Donkers, Maastricht University

• Arno Muijtjens, Maastricht University

• Cees Van der Vleuten, Maastricht University