The Effectiveness of Instruction Time on Science Achievements in G7 and East-Asian Countries

• In Event: Poster session IV
  In Poster Session: Poster session 4

Proposal

Background: Programme for International Student Assessment (PISA) is one of the most popular public available educational data assessing various aspects of 15-year-olds’ education across the globe every three years since 2000 with a focus on mathematics, science and reading on a rotational basis. Up to now, 6 waves of PISA data are publically available. There is a dramatically increasing trend of publications on PISA related research (Hopfenbeck et al., 2018) during the 2000 to 2014 period peaked with 103 publications in 2014.
Among the 72 countries and economies participated in 2015 PISA, the majority of the East-Asian economies are top performers in science and mathematics, with Singapore achieving the highest levels of performance in all three subjects: science, mathematics and reading. In the PISA 2015 report (OECD, 2018), Singapore, Japan, Estonia, Chinese Taipei, Finland, Macao (China), Canada, Vietnam, Hong Kong (China), B-S-J-G (China), Korea are the top 11 performing countries in science, 7 of which are East-Asian economies. Over 25% of the 15-year-olds in 4 of the 7 East-Asian countries (B-S-J-G, Hong Kong, Singapore and Chinese Taipei) achieve top performance in mathematics, with the ability to solve problems that require the formulation of complex situations and the use of symbolic representations (OECD, 2018). Additionally, Japan, Canada and Germany are the three Group of Seven (G7) countries with high levels of performance in science, mathematics and reading, and United Kingdom ranked high in science and reading.

Research Question: With East Asian countries high performers in PISA (Jerrim, 2015) and G7 countries the most affluent countries in the world, this research is interested in comparing the effectiveness of science instruction time on science performance in East-Asian and G7 countries and economies.

Data: This study employed the most recent wave available to public assess is the 2015 data, with over half a million students in 72 countries and economies participated. Considering the diverse social and economic background of the data set, 13 East-Asian and G7 countries out of the 72 countries and economies were sampled for further analysis (N = 109,849).
Outcome variable in this study is average of the 10 plausible values of science scores generated by the multiple imputation process. The independent variable is science instruction time. 25 of the hundreds of the country-level, school-level and individual level variables were selected as covariates for the analysis. The covariates were chosen based on empirical analysis and previous studies (OECD, 2012; OECD, 2018; Gamboa & Waltenberg 2012; Martins & Veiga 2010; Perry & McConney, 2010; Lavy, 2015)). The variables are shown as Table 1 in the Appendix.

Data Analysis: Table 2 lists the sample size, average science score and its standard deviation per country; Table 3 lists the average science instruction time (minutes per week) and its standard deviation. Correlations among the variables of interest, as shown in Figure 1, suggest that there is no issue of collinearity. Figure 2 displays the distributions of science score and science instruction time.
Observations were removed from the data set if having weekly mathematics, science or testing language instruction time over 1200 minutes, resulting 6.59% of the outliers removed from the dataset. After removing these observations, 3.27% of the data were missing. Missing data imputation using MICE (Multivariate Imputation by Chained Equations) method (variables in the dataset that are included as predictors are the variables with at least 0.1 correlation with the target variable) was conducted on each of the 13 countries individually.
Employing the 25 covariates in the dataset, propensity scores were computed based on generalized boosting method (GBM). After propensity score estimations, propensity score was added in the regression model as an adjustment. Although regression model is not the optimal approach to generate covariate balance, especially when the overlap is insufficient, it does not require overlapping diagnostics (Stuart, 2010; Dehejia and Wahba, 1999, 2002; Glazerman et al., 2003) , which is very difficult to check for continuous treatment. Using propensity score as covariate adjustment, the relationship between the propensity score and the outcome was specified as a linear relationship, which met the assumption of this propensity score covariate adjustment method (Austin, 2011).

Results: Before and after propensity score adjustment accounting student level and school level covariates, the science instruction time effects on science performance were estimated respectively.
As listed in Table 4 in the Appendix, propensity score adjustment did make a difference on the science instruction time effect. There is a significant positive effect of science instruction time on science performance for each of the 13 countries. However, after propensity score adjustment, (1) the effect is no longer significant for France and Macao; (2) the significance dropped from 0.001 level to the 0.05 level for Hong Kong; (3) the effects are weaker for Germany and Korea after propensity score adjustment. These 5 countries, as reported, all have average science instruction time below 230 minutes per week, which are not among the top 5 of the 13 countries.

Conclusions: Controlling the 25 student-level and school-level covariates, this propensity score analysis employing PISA 2015 dataset indicates that these 13 countries and economies from G7 or East Asia have different patterns regarding the effects of instruction time. The countries with shorter science instruction time tend to have weaker science instruction time effects after propensity score adjustment.
Even though covariates were considered in the propensity score analysis and we have generated some inferences based on the results, these cross-country comparisons do not suggest any causal relationships between science instruction time and science performance. Future research will be focused on improving the propensity score analysis method to achieve better covariate balance and bias reduction.

Author

• Wan Ren, University of Pennsylvania