Improving Evaluations of STEM Programs: An Empirical Investigation of Key Design Parameters

Effective Years: 2020-2024

To improve science, technology, engineering, and mathematics (STEM) outcomes in K-12 classrooms, it is critical to understand the landscape of the STEM learning environment. However, the STEM learning environment is complex. Students are nested within teachers, and teachers are nested within schools (which in turn are nested within districts), which implies a multilevel structure. To date, most empirical research that uses multilevel modeling to examine the role of higher levels on variation in student outcomes does not examine the teacher level. The reason is that for many states, data linkages between students and teachers have been difficult to achieve. However, in the last five years, this situation has improved in many states, which makes this work now possible. This study seeks to further understanding of the STEM learning environment by 1) examining the extent to which mathematics and science achievement varies across students, teachers, schools, and districts and 2) examining the extent to which student, teacher, school, and district characteristics that are found in state administrative databases can be used to explain this variation at each level. This work will support advances in research and evaluation methodologies that will enable researchers to design more rigorous and comprehensive evaluations of STEM interventions and improve the accuracy of statistical power calculations. Broad dissemination of the resulting tools and techniques will provide access through freely available websites, and workshops and training opportunities to build capacity in the field for STEM researchers to design cluster randomized trials (CRTs) to answer questions beyond what works to for whom and under what conditions. This project is funded by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development. Funding is also provided by the Discovery Research K-12 program (DRK-12), which supports the research and development of innovative resources, models, and tools in order to enhance STEM learning and teaching by pre-K-12 students and teachers.

This project will contribute to 1) describing and explaining the landscape of the STEM learning environment, an environment which includes students, teachers, and schools, and 2) applying this empirical information in the design of STEM intervention studies to enable researchers to extend beyond the usual questions about if the intervention works and for which types of students or schools. By adding teacher level variables, this analysis would account for key teacher characteristics that may moderate the treatment effect. The research will also increase the efficiency in the design of CRTs of STEM interventions. Specifically, the findings from this study will improve the internal validity and cost-efficiency of evaluations of STEM interventions by increasing the accuracy of estimates for the full range of parameters needed to conduct power analyses, particularly when the teacher level is included. The high cost associated with CRTs makes it critical to plan accurate trials that do not overestimate the required sample size, and hence cost more than necessary, or underestimate the sample size and thereby reduce the potential to generate high-quality evidence of program effectiveness. Including the teacher level in intervention studies, a critical level in the delivery of any intervention, will allow for more testing of teacher characteristics that may moderate intervention effects.

This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.