Advisor
Hutchinson, Susan R.
Committee Member
Moren, Amanda J.
Committee Member
Rue, Lisa A.
Committee Member
Olmos, Antonio
Department
Applied Statistics and Research Methods
Institution
University of Northern Colorado
Type of Resources
Text
Place of Publication
Greeley (Colo.)
Publisher
University of Northern Colorado
Date Created
8-1-2013
Genre
Thesis
Extent
247 pages
Digital Origin
Born digital
Description
Increasing demands for design rigor and an emphasis on evidence-based practice on a national level indicated a need for further guidance related to successful implementation of randomized studies in education. Rigorous and meaningful experimental research and its conclusions help establish a valid theoretical and evidence base for educational interventions and curricula. The validity of findings derived from an experimental design largely depends on the quality of the randomization and the study implementation. This study’s purpose was to systematically examine how the magnitude and type of typical randomization problems affected study results. I used secondary data from a randomized national study, the Early Head Start Research and Evaluation Project, to examine the manipulated effects of threats to randomization on selected child developmental outcomes. Data were exposed to 1 of 27 different threat conditions and were compared with randomized data using sensitivity analysis to assess effects on intervention-control balance on covariates and results bias introduced by the threat (Type I and II error, mean percent bias, and effect size differences). The conditions varied by overall sample size (small, medium, large), proportion of the sample disrupted (5%, 15%, 25%), and the type of disruption (allocation bias, noncompliance, differential attrition). The effects of post hoc statistical adjustments, propensity score analysis, and analysis of covariance were also examined. The introduction of imbalance and bias in baseline covariates generally led to bias in the results under threat conditions. The allocation bias scenario was most affected by imbalance under the threat condition, although a high level of imbalance was also introduced in the noncompliance scenarios and a moderate amount in the differential attrition conditions. Baseline imbalance and bias were greatest for the large samples and for the samples that were threatened at the 25% threat level. As expected, the greater the proportion of the sample affected by the threat scenario, the greater the likelihood of baseline imbalance, bias, and biased results. The threat scenario under which the outcomes results were most sensitive was allocation bias, matching the larger baseline imbalance and bias introduced. Examination by sample size indicated a relatively high rate of Type I error was found for the small samples, while the highest rates of Type I and Type II error were among the large samples. Overall bias was highest among the small samples; 35% of the tests were biased either in terms of the significance test, the mean effect size difference, or mean percent bias. The samples that were affected by the largest proportion of threat were the most sensitive to the disruption, again matching the levels of introduced baseline imbalance and bias. For this group, the high mean percent bias (14.3%) was notably higher than the 15% and 5% threat levels. Overall, the adjustment techniques introduced more bias than they corrected. The well-respected randomized design is susceptible to any number of design threats which, depending on the circumstances, might bias effect estimates of interest. Implications for researchers, regardless of study sample size, include measuring sufficient baseline covariates to conduct balance checks, preventing design threats by closely monitoring research practices, and generally using various means, such as literature review and replication, to cross-check findings. Statistical adjustment methods appropriate to the threat type are warranted when bias is likely to be present. The reliance on randomization to prevent all internal validity problems should be in direct proportion to efforts taken to maintain the design’s integrity.
Degree type
PhD
Degree Name
Doctoral
Language
English
Local Identifiers
Green_unco_0161D_10240
Rights Statement
Copyright is held by author.
