Publications
This data article includes information on institutional data at a large public research university in Southern California. In particular, data on undergraduate student enrollments in online and face-to-face courses during summer terms from 2014 to 2017 cumulating in 72,441 course enrollments from 23,610 undergraduate students in 433 courses is provided. This data includes additional information on the statistical models examining factors influencing student enrollment by course modality and the associations of course modality with course grades. This includes descriptive data and data derived from multi-level logistic regression analyses and multi-way fixed effects linear regression analyses. This data article is associated with the article Effects of course modality in summer session: Enrollment patterns and student performance in face-to-face and online classes [1]. (c) 2020 The Authors. Published by Elsevier Inc. This is an open access article under the CC BY license (http://creativecommons. org/licenses/by/4.0/).
Using Clickstream Data to Measure, Understand, and Support Self-Regulated Learning in Online Courses
The ability to regulate one's own learning is essential for success in online courses. Recent efforts have used clickstream data to create timely, fine-grained, and comprehensive measures of self-regulated learning (SRL) in online courses in an attempt to shed light on the process of SRL and to improve the identification of students who lack SRL skills and are at risk of low achievement. However, key questions remain: to what extent do these clickstream measures correspond to traditional self-reported measures about specific SRL constructs? Do these clickstream measures provide more information than existing self-reported measures in predicting course performance? This study used the clickstream data collected from a learning management system to measure two aspects of SRL: time management and effort regulation. We found that the clickstream measures were significantly associated with students' self-reported time management and effort regulation after the course. In addition, these clickstream measures significantly improved predictions of students' performance in the current and subsequent courses over predictions based on self-reported measures alone. These results provide evidence for the validity of the clickstream measures and guide the use of clickstream data to understand the process of SRL and identify students who might not be well served by taking classes online.
Informed by cognitive theories of learning, this work examined how students' self-reported study patterns (spacing vs. cramming) corresponded to their engagement with the Learning Management System (LMS) across two years in a large biology course. We specifically focused on how students accessed non-mandatory resources (lecture videos, lecture slides) and considered whether this pattern differed by underrepresented minority (URM) status. Overall, students who self-reported utilizing spacing strategies throughout the course had higher grades than students who reported cramming throughout the course. When examining LMS engagement, only a small percentage of students accessed the lecture videos and lecture slides. Applying a negative binomial regression model to daily counts of click activities, we also found that students who utilized spacing strategies accessed LMS resources more often but not earlier before major deadlines. Moreover, this finding was not different for underrepresented students. Our results provide some initial evidence showing how spacing behaviors correspond to accessing learning resources. However, given the lack of general engagement with LMS resources, our results underscore the value of encouraging students to utilize these resources when studying course material.
Addressing high demand for developmental math instruction and low rates of successful completing of the developmental coursework, with cost and space constraints, has been an ongoing challenge for post-secondary institutions. With advances in online instructional technology, particularly those based on artificial intelligence, web-based instruction is increasingly considered as a way to alleviate these burdens. This is among one of the first efforts that uses a quasi-experimental design to compare the academic outcomes of students who take a developmental mathematics course in a blended setting that combines face-to-face instruction with an online intelligent tutorial system, ALEKS, to the academic outcomes of students who take the same course in a fully online setting. Results suggest that students receiving online-only instruction perform worse on the final exam and receive lower course grades. However, a cost-effectiveness analysis suggests that fully online instruction has both a lower cost per student enrolled and a lower cost per student passing the course.
Time management skills are an essential component of college student success, especially in online classes. Through a randomized control trial of students in a for-credit online course at a public 4-year university, we test the efficacy of a scheduling intervention aimed at improving students' time management. Results indicate the intervention had positive effects on initial achievement scores; students who were given the opportunity to schedule their lecture watching in advance scored about a third of a standard deviation better on the first quiz than students who were not given that opportunity. These effects are concentrated in students with the lowest self-reported time management skills. However, these effects diminish over time such that we see a marginally significant negative effect of treatment on the last week's quiz grade and no difference in overall course scores. We examine the effect of the intervention on plausible mechanisms to explain the observed achievement effects. We find no evidence that the intervention affected cramming, procrastination, or the time at which students did work.
This quantitative study examines the impact of a three-week online organic preparatory course for chemistry undergraduates that is designed to improve student performance in the subsequent organic chemistry course series (N = 1,289). Organic chemistry often serves as a gatekeeper for students pursuing careers in science, technology, engineering, or mathematics (STEM). Because many students are underprepared for the rigorous organic chemistry series, and consequently are at greater risk of failing it, an online preparatory course was offered that emphasized topics that students frequently struggle with when they enter organic chemistry. The average treatment effects of participation in the online preparatory course on students' subsequent organic chemistry course grades were analyzed utilizing inverse-probability weights with regression adjustment. The analyses indicate that participation in the online preparatory course led to an improvement in subsequent organic chemistry course performance of approximately one-third of a letter grade (e.g., C+ to B-). Notably, students typically at-risk in college environments (i.e., low-income students, first-generation college students, underrepresented minorities) showed commensurate gains when compared to their non-at-risk counterparts. Consequently, this study provides an example of a low-cost intervention that can increase student learning and achievement in organic chemistry. In addition, this study contributes to the nascent research base that examines more distal effects of online course participation.
Conducted in two sections of an introductory chemistry course, the current study assesses the impact of a partially flipped course compared to traditional lectures on student academic performance, motivation, and perceptions. Although the partially flipped course had little impact on student final exam performance in the current course, it had an overall positive effect on student grades in a subsequent course with presence of interaction effect favoring students with lower high school GPA. By implication, the partially flipped course structure has the potential to bridge the achievement gap over time. Similarly, flipped instruction had an overall positive effect on end-of-quarter student motivation, and academically weaker students showed relatively higher motivation increases. Treatment students rated the flipped course much more positively regarding instructional clarity, instructor quality, and course quality. Compared to the student reflections received in our previous study, negative comments were much less in scope and severity in the current study, owing to a gentler approach for introducing flipped instruction. The gentler approach might allow student to adapt to the format over time. Additionally, greater accountability due to increased assignments of the pre-class work contributed to higher student preparation and improved perceptions.
With the nationwide emphasis on improving outcomes for STEM undergraduates, it is important that we not only focus on modifying classroom instruction, but also provide students with the tools to maximize their independent learning time. There has been considerable work in laboratory settings examining two beneficial practices for enhancing learning: spacing and self-testing. In the current study, we examine biology students' study practices, particularly in the context of these two behaviors. We specifically investigate whether a light-touch study skills intervention focused on encouraging spacing and self-testing practices impacted their utilization. Based on pre- and post-course surveys, we found that students report utilizing both beneficial and ineffective study practices and confirm that usage of spacing and self-testing correlates with a higher course grade. We also found that students in the section of the course which received the study skills intervention were more likely to report continued use or adoption of spacing and self-testing compared to students in control sections without the intervention. Surprisingly, we found that underrepresented minorities (URMs) under-utilize self-testing, and that our intervention helped to partially ameliorate this gap. Additionally, we found that URMs who reported self-testing earned similar course grades compared to non-URMs who also self-tested, but that there was a much larger drop in performance for URMs who did not self-test relative to non-URMs who also did not selftest. Overall, we would encourage instructors to dedicate class time towards discussing the merits of beneficial study practices, especially for students that have historically underperformed in STEM disciplines.
Massive Open Online Courses (MOOCs) have the potential to democratize education by providing learners with access to high-quality free online courses. However, evidence supporting this democratization across countries is limited. We explored the question of MOOC democratization by conducting cross-national comparisons of gender differences in the enrollment in and completion of science, technology, engineering, and mathematics (STEM) MOOCs. We found that while females were less likely than males to enroll in STEM MOOCs, they were equally likely to complete them. Further, a higher probability to enroll in STEM MOOCs and smaller gender gaps in STEM MOOC enrollment and completion were found in less gender-equal and less economically developed countries.
Representation and prediction of student navigational pathways, typically based on neural network (NN) methods, have seen their potential of improving instruction and learning under insufficient human knowledge about learner behavior. However, they are prominently studied in MOOCs and less probed within more institutionalized higher education scenarios. This work extends such research to the context of online college courses. Comparing student navigational sequences through course pages to documents in natural language processing, we apply a skip-gram model to learn vector embedding of course pages, and visualize the learnt vectors to understand the extent to which students' learning pathways align with pre-designed course structure. We find that students who get different letter grades in the end exhibit different levels of adherence to designed sequence. Next, we fit the embedded sequences into a long short-term memory architecture and test its ability to predict next page that a student visits given her prior sequence. The highest accuracy reaches 50.8% and largely outperforms the frequency-based baseline of 41.3%. These results show that neural network methods have the potential to help instructors understand students' learning behaviors and facilitate automated instructional support.


