STEM Learning and Learning Environments  

A Statistical Framework and Tools for Planning Multilevel Randomized Cost-Effectiveness Trials

Effective Years: 2020-2026

This research project is supported by the EHR Core Research (ECR) program, which supports work that advances fundamental research on STEM learning and learning environments, broadening participation in STEM, and STEM workforce development. There continues to be increasing demand for more comprehensive assessments of the impacts of STEM education programs, policies, and practices. In general, cost-effectiveness studies are designed to address this demand by evaluating the comparative performance of alternative programs in terms of both their effectiveness and the cost of producing those effects. Current assessments have dealt almost exclusively with program effects. New and more comprehensive assessment tools are needed to detail not just the effects of STEM programs but also the net cost of producing those impacts across the many organizational levels take make up school systems (classrooms, schools, and districts). This research develops a statistical framework and software that critically enhance the design of multilevel cost-effectiveness studies in STEM areas (estimating and separating costs across these levels). The purpose of this study is fourfold: (1) to develop a statistical framework that guides the design of cost-effectiveness studies in terms of power, sample sizes, and minimum detectable effect size (MDES); (2) to assess and substantiate the precision and utility of the resulting formulas through Monte Carlo simulations; (3) to implement the costing formulas in an accessible and user-friendly software program, PowerUp!. In effect generating a new updated version of the software that includes the capacity for multilevel cost effectiveness; and (4) to develop an empirical catalogue of parameter values to guide researchers in the judicious implementation of these analyses. The scope of this project includes a broad range of multilevel designs, including ten of the most common multilevel research designs, and extends them to sensibly include cost effectiveness analyses. Conceptually, these designs vary along three primary dimensions: (1) the number of cluster levels (classrooms, or schools); (2) the level of program assignment; and (3) the level of cost information available. The statistical framework and tools for the effective and efficient design of multisite randomized trials will explicitly address each of these dimensions. This award reflects NSF's statutory mission and has been deemed worthy of support through evaluation using the Foundation's intellectual merit and broader impacts review criteria.