INVALSI big data come to school

• In Event: Roundtable Session (Tuesday II)
  In Refereed Round-Table Session: Assessment, accountability and teacher policies. International and national perspectives and developments applied to ILSAs

Proposal

INTRODUCTION
National Institute for the Evaluation of the System of Education and Training (INVALSI) conducts every year a large-scale survey to measure students’ skills in Italian language and mathematical knowledge and to provide an image of the school system’s progress. INVALSI provides data to different users: the official ministerial evaluation, stakeholders and public community, researchers and other data users and schools. At the beginning of every new school year, INVALSI returns each school its results of all the classes involved in the survey analysed in order to provide tools for school improvement and for a self-evaluation in a comparative perspective. The aim of this paper is to show the procedure proposed by INVALSI for the estimation of school value added and how the elaboration conducted on this data generate an easy and actionable information.
Data provided for schools: school and class details.

The first large scale survey was in 2007/08, consisting of a test at the end of the 8th grade which was included in the final school cycle exam. Currently, as shown in blue in Table 2, INVALSI every year involves 2nd, 5th, 8th and 10th grade students (next school year 13rd grade) in its Italian, Mathematics and English (Reading and Listening) standardized tests. All the published data (used for the annual reports ) and territorial benchmarks are computed on a school sample survey, which follow an administration protocol realized to ensure data reliability. This year INVALSI introduced the Computer Based Tests (CBT) for the 8th and 10th grades, and some new features are in progress like a new test for the 13th grade students, which is the end of the last school cycle.

INVALSI data involved on May 2018 more than 100.000 classes of four school grades and are returned to each school around September/October of the same year through their personal area available on our website . This platform is accessible by personal login: in a unified web portal all different school profiles (from school manager to single teachers) can dispose of national and international assessment data of the last four school years, with tutorials for the correct interpretation of data and figures.

Data returned to schools cover mainly these aspects:
* the general learning performances for all classes and the whole school, compared with territorial benchmarks (regional, macro regional, national) and with the average performances of ‘similar’ classes/schools comparable on the basis of social-economic and cultural context
* variance decomposition of performances social-economic and cultural context between classes
* student’s distribution by learning levels, according to the test performance
* microdata (structural data, Italian and Maths answers and scores, for each single student)
* school value added.

SCHOOL VALUE-ADDED ESTIMATION
The school value added is an indicator of the school contribution to student learning after controlling for the individual-specific variability among students. The school value added is estimated using the multilevel model in order to account for the hierarchical structure of the data, in which students are nested within school. Thus, Y_ij, the academic performance of the student i of the school j is modelled as followed:

Y_ij=β_0+βX_ij+u_j+ϵ_ij

where β_0 is the intercept of the model, β is the vector of fixed-effects parameters, X_ij represents a vector predictors, u_j is the random effect associated with the school j with mean zero and variance σ_u^2 and ϵ_ij is the individual normal error with mean zero and variance σ_ε^2.

The term u_j allows to model the dependence among the students that are attending the same school and captures the school value added, an unobservable quantity that characterizes the school j and is shared by all its students estimated by the residual component of the multilevel model.

For the estimation of the school value added, a fundamental element is the inclusion as predictors of only the exogenous variables, that are those variables which value is not determined in the system being studied. The Table 2 reports the list of the variables, both at student and school level, included in the model.

INVALSI returns to schools the estimation of the school value added categorized in five classes using as threshold the national (or regional) mean of the school value added plus or minus 1 or 2 standard deviations. After controlling for the individual-specific variability among students, the school effect is categorized as shown in the Table 3. An example of distribution of school effect about grade 5 and grade 8 is reported at national level in Table 4 and Table 5; but it is more interesting to observe the same effect in different Italian geographical areas (Table 6 and table 7) where it is possible to note that the South has a very difficult situation especially in Grade 8 with the most percentage of schools in the Negative position of Value added.

This procedure allows to obtain an indicator of the school efficacy that is modelled starting from the whole INVALSI data but can be easily interpreted and is usable by each single school. The indicator permits to give at each school a measure of the school self-efficacy respect to geographical benchmarks. Moreover, the school value added together with the observed score in INVALSI standardized test (Graph 2) allows schools to identify their weaknesses on which plan improvement actions.

CONCLUSIONS
INVALSI assessments data gained a scientific and institutional recognition and become an increasingly common tool adopted by schools at local and institutional level: for example, they are used to guide parents’ school choices and attract registrations or as a qualification to participate at local projects or at specific regional or national interventions. INVALSI provides tools for school improvement and promotes a self-evaluation in a comparative perspective. Indeed, schools are using this kind of indicators to plan improvement actions.

Authors

• Michele Cardone, INVALSI

• Patrizia Falzetti, INVALSI

• Chiara Sacco, INVALSI