MakEval: Developing a Set of Tools to Evaluate the Benefits of Making

• In Event: Measuring Making: Methods, Tools, and Strategies for Capturing Learning, Participation, and Engagement in Maker Activities

Abstract

While researchers advance work in understanding various aspects of makerspaces and the increase of making in education, we lament the lack of robust research tools that can be used for research and evaluation of these experiences. While we have used surveys and observation protocols that exist for evaluating STEM-learning in other projects (Maltese & Harsh, 2015; Simpson, Burris & Maltese, Forthcoming), the reality is that making is rather different from many other contexts and a suite of maker-specific tools is warranted.

This issue is particularly salient when educators ask us to suggest tools for them to use to gather data to justify their use of making to administrators and funders. Interestingly, this issue exists for single teachers and small museum staff as much as it does for science centers with full evaluation teams - no one has had the time to focus on developing tools while also getting their programming started or expanded to meet the needs of the people they serve. As maker programs expand around the country, the need for tools to evaluate the elements of the program are of critical importance to judge impact. Basic research and evaluation sets the foundation for understanding key program elements to scale as these programs grow.

What we will present is our work toward creating a suite of tools, including surveys, assessments and observation protocols, that provide educators, researchers and program administrators with information to evaluate maker programs/experiences with youth. Based on data collected from surveys and interviews of maker educators from formal and informal contexts, we selected a set of five key targets as our initial targets for evaluation, including: creativity, engagement, problem solving, attitudinal changes, agency/independence, and involvement in science and engineering practices.

Although a single tool that could answer the major questions that we (and others) have would be ideal, that is unrealistic and most “common instruments” are often too blunt to be of value. As we advance this work we are guided by the ideal that there are a number of features that we need to address with each tool and across the set:
--Ability to judge changes over time
--Standardization across contexts
--Useful for providing both formative and substantive feedback
--Must be age/developmentally appropriate
--Gather school/community data to complement
--Useful for educators/admins who have limited time/resources to complete
--Associated with convincing reliability and validity evidence
Throughout all this we are seeking to find an acceptable balance between richness of findings and the pragmatic needs of those who have limited time to determine if making is “working” for their youth. We will discuss our progress toward finding, modifying or developing tools to assess our key constructs while also meeting the critical features to make the tools useful to the community. This will include reviews of existing tools, description of modification of tools or creation of new tools to serve our needs. Additionally, we will discuss early phases of data collection and analysis with the suite of tools in a variety of making contexts.

Authors

• Adam V. Maltese, Indiana University

• Amber Simpson, Binghamton University - SUNY

• Jean J. Ryoo, UCLA

• Alice C. Anderson, Minneapolis Institute of Art

• Meihua Qian, Clemson University