In a study of 370 postsecondary students in electronics, engineering, and other science classes, we investigated collaborative problem-solving (CPS) skills that best predict performance at individual levels in an online electronics environment. The results showed that while monitoring was a consistent predictor across levels, other skills such as executing, sharing information, planning, and maintaining communication each predicted individual performance at one or more levels of the task. The availability of background data on students' content classes and associated content knowledge to analyze the model results can help identify possible cues for instructors across domains to help students improve specific CPS skills to achieve high performance in activities conducted in collaborative learning environments. © 2024 Educational Testing Service.

This study discusses the development of a basic electronics knowledge (BEK) assessment as a pretest activity for undergraduate students in engineering and related fields. The 28 BEK items represent 12 key concepts, including properties of serial circuits, knowledge of electrical laws (e.g., Kirchhoff's and Ohm's laws), and properties of digital multimeters. This paper first discusses a psychometric evaluation of the BEK assessment to understand its basic measurement properties and to examine various group-level differences based on demographic, institutional, and instructor characteristics. Subsequently, the relationship between BEK scores on the 23 retained items and performance on an existing complex collaborative simulation-based electronics task is discussed. Results demonstrated that basic content knowledge alone may not be sufficient for students to demonstrate knowledge of electronics skills on more complex tasks. The research also carries great importance given ongoing concerns about improving the overall state and diversity of the engineering workforce and its associated pipeline to meet the demands of the national economy. © 2020 Educational Testing Service

Collaborative problem solving (CPS) is a complex construct comprised of skills associated with social and cognitive dimensions. The diverse set of skills within these dimensions make CPS difficult to measure. Typically, research on measuring CPS has used highly constrained environments that help narrow the problem space. In the current study, we applied the in-task assessment framework to support the exploration of CPS skills at a deep level in an open digital environment in which three students worked together to solve an electronics problem. The construct of CPS was defined in depth prior to the implementation of the environment through the development of a complex, hierarchical ontology. The features from the ontology were identified in the data and four theoretically-grounded profiles of types of collaborative problem-solvers were produced - high social/high cognitive, high social/low cognitive, low social/high cognitive, and low social/low cognitive. Results showed that students in the low social/low cognitive profile group demonstrated poorer performance than students in other profile groups. Further, having at least one high social/high cognitive member in a team facilitated performance. This study offers groundwork for future studies in measuring CPS with an approach suitable for less constrained collaborative environments.

Collaborative problem solving (CPS) has been deemed a critical twenty-first century competency for a variety of contexts. However, less attention has been given to work aimed at the assessment and acquisition of such capabilities. Recently large scale efforts have been devoted toward assessing CPS skills, but there are no agreed upon guiding principles for assessment of this complex construct, particularly for assessment in digital performance situations. There are notable challenges in conceptualizing the complex construct and extracting evidence of CPS skills from large streams of data in digital contexts such as games and simulations. In the current paper, we discuss how the in-task assessment framework (I-TAF), a framework informed by evidence-centered design, can provide guiding principles for the assessment of CPS in these contexts. We give specific attention to one aspect of I-TAF, ontologies, and describe how they can be used to instantiate the student model in evidence-centered design which lays out what we wish to measure in a principled way. We further discuss how ontologies can serve as an anchor representation for other components of assessment such as scoring rubrics, evidence identification, and task design.