STEM Learning and Learning Environments  

Assessing Complex Collaborative STEM Learning at Scale with Epistemic Network Analysis

Effective Years: 2017-2022

This project was submitted in response to EHR Core Research (ECR) program announcement NSF 15-509. The ECR program of fundamental research in STEM education provides funding in critical research areas that are essential, broad and enduring. EHR seeks proposals that will help synthesize, build and/or expand research foundations in the following focal areas: STEM learning, STEM learning environments, STEM workforce development, and broadening participation in STEM. The ECR program is distinguished by its emphasis on the accumulation of robust evidence to inform efforts to (a) understand, (b) build theory to explain, and (c) suggest interventions (and innovations) to address persistent challenges in STEM interest, education, learning, and participation. In this project, researchers will develop a statistical analysis technique for measuring how people learn. With prior NSF funding, members of the team created epistemic network analysis (ENA), a technique for creating network models of complex and collaborative thinking in science, technology, engineering, and mathematics. ENA is being used by more than 40 researchers at 19 universities to answer a wide range of research questions in learning sciences, cognitive neuroscience, engineering education, environmental science education, medical and surgical education, and history of science. The proposed research and development will create an online toolkit that lets researchers upload audio, video, text, or log-file data, automatically transcribe the audio data, develop and validate automated codes using supervised natural language processing tools, and produce ENA models. This will make it possible for researchers analyze data on how people learn without requiring simultaneous expertise in automated transcription, data segmentation, coding, and network modeling. It will also make it possible to conduct analyses of learning using the large volumes of data that are currently generated by online learning tools, significantly expanding capacity for research on learning. In this project, the research team will conduct fundamental research on learning in science, technology, engineering, and mathematics (STEM) by developing and extending a theory-based statistical analysis technique for using network analysis to model complex and collaborative STEM thinking (CCST). With prior NSF funding, members of the team created epistemic network analysis (ENA), a technique for creating dynamic models of CCST. Data on CCST typically come in one of two forms: Video and audio recordings of interactions from classrooms and workplaces, or log files from online interactions. Data in audio or video form must be transcribed. Text data from recordings or log files have to be coded, or annotated to indicate what elements of CCST are present in the data, and where those elements are located. With existing tools, researchers must complete these steps by hand, or use their own techniques for automated transcription and coding. However, many skilled CCST researchers are not simultaneously experts in the sciences of automated transcription, automated identification, validation, and application of codes, and network analysis. The proposed research and development will create ENAlysis, an online toolkit that provides a seamless, automated analysis pipeline from raw data to final results. The project will thus develop innovative methods for measuring STEM learning and expand access to a proven, theory-based technique for analyzing CCST. The result will be expanded use of powerful learning analytic techniques to model CCST, which will significantly improve assessment of STEM thinking and learning and inform policy and practice.