Assessment of Predictive Validity of Entrance Teacher Exams in Mexico Using A Two-Methodology Approach: Implications for Teacher Evaluation Reform

• In Event: Poster session I
  In Poster Session: Poster session 1

Proposal

In order to improve education quality many countries have implemented teacher evaluation systems that aim to improve teaching quality (Isoré 2009, Xibin 2003). Mexico is one of these countries aiming to improve the quality of teachers, and consequently education, by establishing a series of reforms largely based on high-stakes summative teacher evaluation processes. This reform has also opened the doors to teaching candidates from the general public (graduated from schools other than normal or pedagogical schools) who desire to pursue a teaching career. Suitable teachers are identified through the teacher entrance exam, and evaluated again after two years of teaching to ensure their performance justifies the continuation of their appointment.
Studying contributions to validity of entrance evaluations or exams is particularly relevant in high-stakes testing given that the interpretation and uses of scores leads directly to decisions that impact the lives of test takers. Thus, it is important to ensure that the new high-stakes evaluation system implemented in Mexico is working to enable the selection of suitable teachers into the education system. Accordingly, I pose a research questions that focuses on measurement properties of the assessments: Is there evidence that contributes to the predictive validity of the assessments on teacher performance after two years of teaching? Can entrance exam scores discriminate between the suitable and not suitable classification of teachers that result from the evaluation?
In Mexico, all available teaching positions are subject to this entrance evaluation. Then, according to the results obtained by teacher candidates in the two assessments teaching positions are assigned in descending order: from the highest to the lowest scores, as long as teachers are qualified as suitable. Thus, in theory, through this new implemented evaluation system, only those teacher candidates that obtain the “best results” would enter the education system (CNSPD 2014). In practice, however, if there are not enough suitable teachers, non-suitable teachers are assigned to a “limited-time” teaching position, which could have implications in the validity of these assessments, which are explored in this paper.
I use two distinct – yet complimentary – methods to explore the predictive validity of the trance exam comparing entrance and performance results after two years of teaching of the first two cohorts that entered the education under this reform (2014-2015 and 2015-2016); and exploring the relationship between global performance after two years of teaching and scores obtained upon entrance to teaching. First, to investigate this correlation, I use linear regression analyses to explore the extent to which the classification of lower and higher scores from the entrance exam compare to the suitable and not suitable classification results. Second, using the Bayesian predicted scores of teacher candidates that did not entry the teaching system, I compare the differences in performance after two years of teaching for all candidates – those who passed and did not pass the entrance exam. Given that I cannot observe the scores obtained after two years of teaching for teacher candidates who did not pass the entrance exam, these scores are not ignorable and missing not at random (NMAR). Thus, following the recommendations by Muthén et al. (1987) and Collins et al. (2001), I use a Bayesian multiple imputation (BMI) approach to generate such scores. Social scientists are often restraint to the use of datasets where the outcome of interest is unobserved and systematically different for the cases with observed and unobserved outcomes, wages are unobserved for people who are unemployed or out of the labor force and test scores are unobserved for non-takers, for example (Garlick and Hyman, 2016). Consequently, these data can be found in selective environments. In these contexts, data and methods recommendations to address such missingness include assessing the presence of non-random sampling using multiple logistic regression; mean differences on the study’s variables between those who responded and did not respond to the subsequent data collection; restricting or enhancing variances, and changes in relationships among variables due to attrition (Goodman et al., 1996). However, imputing missing values with the mean will truncate the variance of the variable as well as its covariance with other variables (Peugh & Enders, 2004). An example of this issue is the case of the SAT and college performance where less than half of public high school students nationwide take the ACT or SAT, limiting policymaker and researchers from using these scores as a true measure of college-readiness population (Garlick and Hyman, 2016). The Bayesian approach is a theoretically appealing framework for developing imputation methods because it specifies an explicit model for variables with missing values, conditional on the fully observed variables and some unknown parameters, a prior distribution for the unknown parameters, and a model for the missing data mechanism, which does not need to be specified under an ignorable missing data mechanism (Rubin, 1976). This explicit model then generates a posterior predictive distribution of the missing values conditional on the observed values (Raghunathan et al., 2001). Epidemiological and prognostic factors studies in medicine often make use multiple imputation to estimate models with missing data (Royston and White, 2011, for example). In this paper, I follow the same approach by imputing teaching performance outcomes upon entrance by imputing data using a normal-normal distribution. While these types of models are not often used in the education field, missing data are a common occurrence in educational assessment.
This paper contributes to assess the predictive validity and measurement properties of teacher entrance assessments in Mexico by exploring the extent to which teacher entrance scores predict teacher performance after two years. In addition, my research contributes to the educational assessment field by extending the properties of predictive validity beyond the scope of candidates’ selection; i.e., teachers that did not pass the entrance evaluation, while current studies have only studied predictive validity based on selected candidates alone. Finally, my study introduces to educational measurement research a novel method to analyze incomplete data, especially when this data is not ignorable and MNAR.

Author

• Mariana Barragan Torres, UCLA