Poster #82 - Online instuction better than offline? Evidence from Learning Process Data of a Large Sample

• In Event: Poster Session 19

Abstract

Due to the COVID-19 pandemic, a growing body of research has compared the effectiveness of online and offline instruction. However, most of these studies examined the effectiveness merely based on students’ test scores, but rarely investigated students' learning process data. Therefore, the current study employed process data of students’ homework to examine the effects of online-instruction on math achievement and learning efficiency.

From March to June in 2022, all schools in Shanghai, China, switched from offline to online instruction because of the pandemic, which created a natural cohort of students receiving online instruction only. The cohort of students in the previous year 2021 comprised as a natural comparison group who received offline instruction during the corresponding time period. The current study included a sample of 11910 fifth graders (48.5% girls) in 2021 ("offline-instruction group"), and 13747 fifth graders (48.2% girls) in 2022 ("online-instruction group"). Both groups took the same two Math assignments one month apart from each other in their academic year. These two Math assignments were standardized homework developed by the local department of education (alphas = .801 and .802 for each assignment).

First, the Generalized Partial Credit Model was used to calculate the assignment grades as the indicators of student's academic performance. Students’ learning efficiency was assessed by the assignment grades obtained per time.
Next, a 3-level Hierarchical Linear Model (HLM) was used to analyze the effect of instruction mode (coded as 0 for offline and 1 for online) on students’ academic performance while controlling the effect of the baseline performance, gender, and school type (level-1: students; level-2: classes; level-3: schools). Similarly, we build a 3-level HLM to analyze the effect of instruction mode on students’ learning efficiency.

The results of 3-level HLM predicting academic performance are shown in Table 1. Based on Model 1, the ICC estimates indicated that 83% of the variance was explained at level-1, 11% explained at level-2, and 6% explained at level-3. This result provided support for using a 3-level model. Results of Model 3 showed that compared to offline instruction, online-instruction exerted a stronger impact on students’ academic performance while controlling the effect of gender school type, baseline performance, and class-level and school-level characteristics.
The results of 3-level HLM predicting learning efficiency are shown in Table 2. The result of Model 1 provided support for using a 3-level model: 87% of the variance was explained at level-1, 8% explained at level-2, and 5% explained at level-3. Similarly, online-instruction can positively impact learning efficiency while controlling the effect of gender, school type, baseline efficiency, and class-level and school-level characteristics.

In sum, online-instruction has a positive influence on students’ academic performance and learning efficiency. The effectiveness of online instruction might be resulted from that students could study the online lessons repeatedly based on individual needs, and may receive more supervision from their parents during online lessons at home. The current study provided evidence of the effectiveness of online instruction which may help educational policy makers better understand the role of online-instruction in educational practice.

Authors

• Fangyan Cheng, 17 Education and Technology Group
  Presenting Author

• Manjie Wang, 17 Education and Technology Group

• Yan Li

• Yingying Liu

• Tairu Liu