Flexibility in Design, Outcomes and Analysis in “Evidence- Based” Drug Prevention Research: The Case of the Midwestern Prevention Project

Flexibility in study designs, definitions, outcomes and analytic models increases the chance that the results reported in an empirical study are untrue. Such flexibility in methodological and analytic practices has been observed in evaluations of a number of drug prevention programs that appear on lists of evidenced-based programs. The current paper examines the evidence base pertaining to one of the most well-established drug prevention programs, namely the Midwestern Prevention Project (MPP; also known as Project STAR). Specifically, it examines the results reported from the quasiexperimental evaluation of the MPP that was conducted in Kansas City and the experimental replication conducted in Indianapolis, Indiana, and the data analysis practices used in producing these findings. It shows there is considerable analytical flexibility evident in the published accounts from these two evaluations studies, notably in the samples used in the data analyses and the manner in which outcome variables were measured. The implications of this for the MPP’s status as an evidenced-based drug prevention program are discussed, along with the means by which flexibility in analytic procedures in drug prevention research could be reduced. The seemingly widespread existence of such analytic flexibility within the field suggests that confirmation bias is part of the general culture of drug prevention research. This impedes the development of a sound scientific base within the discipline and might even lead it to degenerate into a pseudoscience.