Public-Private Partnerships to drive change in Thailand – an evaluation of Chevron’s Enjoy Science Program

• In Event: From research to practice

Proposal

Chevron’s Enjoy Science Project is a $30 million, five-year, public-private partnership to strengthen Thailand’s competitiveness and innovation by improving science, technology, engineering and math (STEM) education, and technical vocational education and training (TVET) across the country. It is accomplishing this through the professional development of STEM teachers and the rollout of STEM and Technical Vocational Education Hubs nationwide. Convening a range of partners in civil society, academia, the private sector, and the government, Enjoy Science is directly benefiting more than 600 schools and 500,000 students, teachers, principals, government education officials, community members, and workers, providing increased career opportunities and higher wages. This is the largest program of its kind in Thailand. The problem the program seeks to solve is that Thai graduates of secondary and tertiary technical schools do not have the skills (i.e. technical, vocational, etc.) demanded by employers in STEM and TVET related fields, which include Chevron. Students are underemployed and companies are spending a lot of time on training programs and still not realizing ideal productivity levels.
In May 2016, Kenan Institute Asia partnered with Management Systems International (MSI) to provide third party evaluation services to the Chevron Enjoy Science Project, on which Kenan serves as the lead implementing and coordinating agency. The aim of this evaluation is to deliver rigorous and reliable measurements of program effectiveness and sustainability of the Hub model through an innovative model of engagement with academicians and researchers across Thailand. This presentation will focus on midline evaluation results related to the effectiveness of the Enjoy Science program, as well as a discussion of the public-private partnership between Kenan and Chevron, the evaluation consortium with 15 Thai universities led by MSI, and the successes and unique opportunities associated with the evaluation partnership. MSI has learned some critical lessons from baseline and midline experiences with the Enjoy Science evaluation that we are eager to share with peers at CIES, such as what we would do differently knowing what we know now about the project’s frequently shifting implementation plan and objectives. From other CIES panelists and audience members, MSI seeks to understand if and how such evaluation partnerships have been successfully employed in other country contexts, as well as learn more about innovative methods used by other evaluators to assess STEM and TVET programs.
The Enjoy Science evaluation uses a mixed-methods approach relying on both primary and secondary data analyses to establish baseline measurements of Enjoy Science in Thailand between 2015 and 2016 and midline results in 2018 (endline data will be gathered in 2019). Both baseline and midline results will be presented in this paper. Primary data (quantitative and qualitative) documented perceptions of students, math/science teachers and directors on a diverse range of topics, such as students’ perceived competence, teachers’ practices in the classroom, students’ and teachers’ perception on the utility and long-term usefulness of STEM studies. Secondary data included the Ordinary National Educational Test (O-NET) and the Vocational National Education Test (V-NET) exams assessment data; however we faced significant politically sensitive challenges after uncovering issues with validity, reliability, equating and scoring of this data in Thailand. With baseline and midline data in hand for all 3 components of the Enjoy Science program MSI is now positioned to measure the full impact of a landmark program that seeks to transform STEM and TVET education in Thailand.
Midline estimates on quantitative Enjoy Science outcomes will be shared. A difference in difference model was employed for assessments of student achievement and other outcomes. Preliminary analyses for the STEM and STVET components suggest positive impacts on student engagement and skills, likelihood to select STEM related fields of study, and students’ perceptions of teachers’ classroom practices. Direct observations of teaching practices also suggest positive program effects; however, teachers report additional technical supports are needed to fully implement Enjoy Science methodologies, such as the inquiry-based and project-based approach to math and science instruction, in their classrooms. For the TVET component, results related to curriculum relevance, student skills, teacher competencies and the effectiveness and sustainability of the Hub model will be explored. These quantitative findings will be supplemented with qualitative insights related to the context in which Enjoy Science professional development activities take place and stakeholder perceptions on other factors that may influence the effectiveness, relevance and sustainability of the program.
A unique evaluation consortium
This evaluation effort has been given valuable support from a wide range of stakeholders. The Enjoy Science evaluation team consists of evaluation and education experts from MSI and the members of the National Research Team (NRT) — education professors and research assistants from 15 regional universities throughout Thailand. This consortium is uniquely positioned to ensure the relevance and sustainability of the Enjoy Science research agenda in the Thai context. Together we have jointly developed the evaluation design and implementation plan, which includes between 3- 5 impact-level research questions for each program component, as well as a supplemental learning agenda guided by the specific interests and capacities of the NRT. The NRT further recruited research assistants to help carry out the data collection activities, and who contributed valuable feedback throughout analysis and reporting. They played an important role in collecting and consolidating the data. Kenan Institute Asia contributed to the effort by providing support and strategic advisory services, and also assisting with the process of entering data into machine readable formats for reflection, analysis, and dissemination. An MOU with the National Institute of Education Testing Service (NIETS) provided timely access to national O-NET and V-NET data. Validation and interpretation workshops held with education and research experts at the Thailand Research Fund (TRF), Office of the National Economic and Social Development Board (NESDB), and universities provided a critical review of the study design and methodology, and deeply contextualized/localized interpretation of the findings. Moving forward, the Enjoy Science program will continue to rely on these important contributions from these stakeholders.

Authors

• Nitika Tolani, MSI

• Idalia Rodriguez-Morales, Management Systems International

• Sean Kelly, MSI