Measuring post pandemic changes in teaching and learning: A survey of primary Math & English teaching in Rwanda

Group Submission Type: Formal Panel Session

Proposal

Building Learning Foundations (BLF) is a programme of the Rwanda Ministry of Education (MINEDUC) and is funded by UKAid from the British people through the United Kingdom’s Foreign, Commonwealth, and Development Office (FCDO). The programme aims to improve learning outcomes by enhancing the quality of teaching of English and Mathematics at lower primary school level and by strengthening school leadership for learning in all of Rwanda’s government and government-aided primary schools. Programme interventions began in January 2018 and will run until 2023.

BLF baseline assessment was done in 2017 during the start of the programme. A third progress assessment was conducted in the month of October 2021 which related to activities part of the original programme, with teacher observations and pupil assessments of P1-P3 only.

The programme is designed as an intervention over three phases, with each phase covering ten of the 30 districts in the country. As of this third progress assessment, Phase I schools would have participated in BLF activities for approximately three and a half years, Phase II schools for three years, and Phase III schools for two and a half years.

The performance assessments (baseline, progress, midline and endline) are designed to measure whether (and, to some extent, how) desired change outlined in the BLF theory of change is affected through BLF interventions.

The 3rd BLF progress assessment set out to measure teaching and leadership practices as well as pupils’ achievement in P1-P3 English and Mathematics in public and government-aided schools in Rwanda.

Specifically, the progress assessment has three main objectives:

to collect data on BLF’s implementation and results over the past two years and to analyse this in the context of the BLF and Learning for All (LFA) monitoring and evaluation framework and programme logic (log-frame);

to use the analysis and key findings to make recommendations that inform adaptive programming.

Like Baseline and Progress II, Progress III is designed to be nationally representative and therefore robustly comparable to the baseline assessment.

Methodological approach

This mixed-methods assessment designed to determine how well, in comparison to the baseline and previous progress assessment. The programme has met its expected outputs and outcomes. The intention is to build a clear and nuanced picture of what change is or is not taking place and why.

BLF works with all 3100 public and government-aided schools.For this progress assessment, as for the baseline assessment, sampling procedures and sample size calculation ensured a nationally representative sample, with all provinces and districts selected.

The study population covers all the BLF-supported schools and all district and sector education officials and entities benefiting from BLF interventions.

Sampling strategy

A sample of 78 schools across all of 30 districts was computed as the minimum sample which will provide national estimates in terms of learning outcomes, and teaching and leadership practices.

The sample size for schools was determined using schools as the sample units and the two-stage cluster sampling technique was applied with the first sampling stage as the district and the second stage being the pupils to be assessed. The sampling included considerations on power calculations, the expected effect size and the sizes of schools and was verified using the G*Power software. With a two-tail test, the sample was determined considering the following statistical parameters:

An effect size of 0.28,

An α error probability of 0.05,

A power (1-β err prob) of 0.95,

Expected inter-class correlation (ICC, or roh) of 0.05

A design effect (deff), set at 1.26

The sample will have 78 clusters (schools) sampled across all 30 districts in Rwanda.

Teacher samples: Using simple random sampling, three teachers for English and three teachers for Mathematics were selected in each sample school to be observed for teaching practices. The process for selecting teachers for the Aptis test was necessarily different, as the test is conducted at test centres. A sample of 25 English teachers per district, determined through simple random sampling, were used.

Sampling pupils: The second sampling unit is the class, which is selected randomly by the enumerator from all available English and Mathematics classes for each of P1, P2 and P3. The third sampling unit are pupils attending P1-P3 classes. The average ratio of pupils per class in Rwandan schools is 45.

Randomisation is used to select pupils who will be assessed. The same procedure applies for both English and Mathematics progress assessments.

Teacher assessment

The classroom practice development tool asks observers to rate teachers against 12 quality questions covering six pedagogical areas, based on the competencies required by teachers according to international evidence on best practice in addition to the principles outlined in the Competency Based Curriculum and MINEDUC’s inspection standards. There are also two subject specific competencies assessed for each subject and two competencies relating to COVID guidelines. Classroom practice observations were made of 406 teachers, comprising 202 English lessons and 204 Mathematics lessons, in 28 districts across all five provinces.

Pupil assessments

BLF progress assessments assess P1-P3 pupils’ key foundational learning skills in English and Mathematics. The pupil assessments are aligned to the national curriculum and examine the key foundational learning skills targeted by BLF. They were designed to minimise the burden on respondents, especially considering the pupils’ young age, and to make the assessment process more efficient by quickly identifying pupils’ current level of learning. They were also designed to minimise the possibility of a floor effect by capturing pupils’ abilities in multiple dimensions and levels (proficiency is measured overall and by learning domain/skill, with pupils able to demonstrate proficiency at different grade levels: own grade and grade below in Mathematics, any grade P1-P3 in English).

Sub Unit

• General Pool

Chair

• Silas Bahigansenga, Education Development Trust

Individual Presentations

• Measuring changes in teaching & learning of English - Vanessa Komiliades, British Council

• Measuring changes in teaching & learning of Mathematics - Salome Ong'ele, Education Development Trust

• Teacher voices: KAP survey findings - Jean Luc MUSONI, Education Development Trust