Assessing Problem Solving in Technology-Rich Environments: Lessons From the International Assessment of Adult Competencies Survey

• In Event: Assessments for Problem Solving, Literacy, and Learning in Designed, Online Environments

Abstract

Goals: The purpose of this paper is to present the technology-designed environments and main outcomes from an international survey of adults’ Problem Solving in Technology-Rich Environments (PS-TRE) that was conducted as part of the Organization for Economic Cooperation and Development’s “Program for the International Assessment of Adult Competencies” or PIAAC.

PIAAC is the first large-scale survey to investigate adults’ problem solving ability in designed, technology-based environments through direct cognitive assessment tasks as well as self-reports of technology use.

Theoretical framework: The paper introduces the PS-TRE conceptual framework that was designed by the international expert group commissioned by the OECD. The framework considered three core dimensions of problem solving in technology-rich environments: the intrinsic features of the problem solving task, the characteristics of the environment, and the cognitive processes brought to bear by the person. The framework also considered different ways in which information technology affects problem solving: changing the structure of problems (e.g., seeking advice from others about issues of personal relevance), or the means used to solve a problem (e.g., sorting dozens of items using a computer application), or even create new problems (e.g., getting rid of a virus).

Data sources: In its first round (2012), PIAAC was administered in 24 countries with about 5,000 individuals participating in each country. We present the main outcomes of the survey, including a description of the proficiency scale that obtained from the performance data. It is worth noting that across the participating countries, a percentage of participants could not take the computerized problem solving assessment either due to lack of prior experience with computers, failure to complete a core prerequisite test, or unwillingness to take the computer test. We also report these results.

Results: Results from the survey were still being analyzed at the time of submission, and are to be made public in October 2013. A preliminary analysis suggests that problems involving ill-defined goals, multiple steps, a high level of integration and inference, but also the use of multiple technology applications and/or unfamiliar operators were the most difficult. The data were in-line with a primary traits rubric that was created based on the conceptual framework.

In this presentation, we also present the analysis of on-line strategy data extracted from the participants’ computer interaction logs. Analyzing log data as part of a large-scale international survey administered at the participants’ homes poses a number of conceptual and technical issues, such as the identification or computation of relevant operators and their interpretation. A series of case studies involving a selection of PS-TRE items were being conducted at the time of submission. These data shed additional light on the relationship between individuals’ engagement, strategy and problem-solving outcomes.

Significance: Overall the PIAAC PS-TRE survey provides a unique contribution to our knowledge of adults’ abilities to deal with information problems by using computer applications. Directions for future work, in the context of the ongoing evolutions of information technology and technology uses in most societies, will be discussed.

Authors

• Jean-Francois Rouet, CNRS and University of Poitiers

• Anne Britt, Northern Illinois University