Career Readiness Features: Identifying, Defining, and Implementing

• In Event: Developing Career Readiness Indicators: A Focus on Feature Analysis

Abstract

Objective/Purposes
This paper presents the process of identifying, defining, and refining career readiness features through qualitative feature analysis. Linking the context of the Smarter Balanced testing program, feature analysis is a method to characterize items and tasks by components, so that item design, revision, measurement, and instruction may benefit (Authors et al., 2015).

Method/Data Source
To examine career readiness features in Smarter Balanced assessments, we chose two exemplar careers which have been determined by the U.S. Department of Labor as having a “Bright Outlook.” More specifically, the two careers—web developer and emergency medical technician (EMT)—are considered rapidly growing in the U.S., and do not necessarily require a four-year college degree. Drawing from the O*Net database, we identified specific skills, abilities, work activities, and work context features important to these two exemplar careers. Previous work on feature rating schemes as well as Department of Labor career information and data informed the feature selection process. The list of features was refined by considering Smarter Balanced items and blueprints. Subject-matter experts from the areas of college and career readiness, business, and the domain areas of web developer and EMT then examined the feature set list for further refinement, leading to a list of 36 features.
Using practice math and English language arts (ELA) items from Smarter Balanced, a cognitive lab study (N = 17) was conducted with high school students from three schools representing diverse backgrounds. The purpose of the cognitive lab interviews was to validate the feature analysis process and confirm the presence of the career readiness features. In the items, the cognitive lab consisted of both concurrent and retrospective reports (Ericsson & Simon, 1993). In other words, students used a think-aloud process while completing items, and also answered immediate questions about their problem solving and understanding of the items.

Results
Feature ratings by expert raters found between 8 to 13 career readiness features across each of the test items selected for the cognitive lab. Interrater agreement was, on average, 88.8%. Thus, results from the cognitive labs supported the presence of the features in the Smarter Balanced items. Some examples of the features include critical thinking, reading comprehension, deductive reasoning, analyzing data or information, number facility, and processing information.

Significance
These results suggest that the use of feature analysis in math and ELA assessments have the potential for added value to existing assessments with respect to career readiness. Going beyond measuring mathematics and ELA content knowledge is beneficial, especially given the amount of resources dedicated to completing these assessments. One is able to draw career readiness inferences without increasing the testing time burden to students, teachers, or school staff. There is potential impact on policy, curriculum, and the future of assessment as we guide and prepare our students for success in today’s increasingly competitive, knowledge-based world.

Authors

• Ayesha Madni, University of Southern California

• Eva L. Baker, University of California - Los Angeles