First Advisor

Hutchinson, Susan R.

Document Type

Dissertation

Date Created

12-2017

Department

College of Education and Behavioral Sciences, Applied Statistics and Research Methods, ASRM Student Work

Abstract

Purpose: The purpose of this dissertation was to generate observed scores under complex data conditions often found in the real world and (a) investigate error in terms of internal consistency reliability within the Classical Test Theory framework (Cronbach’s a and polychoric ordinal a) and person reliability within Rasch Rating Scale Model (RSM); (b) inform applied researchers about possible relative bias in reliability coefficients when more complex data structures and underlying distributions are encountered; and (c) provide applied researchers a reference from which to interpret their results. Methods: Using Monte Carlo simulation techniques to generate polytomous response choices in single-level and multilevel models, sample reliability coefficients, standard errors of reliability estimates, and levels of absolute relative bias were examined and compared across a range of data conditions, including normal, mixed, and nonnormal distributions and varying sample sizes. Results: The results support taking the structure of the data collected into account during the analytic phase and provide empirical evidence that if data collected for research are dependent on a higher order structure, reliability coefficients in a multilevel model are less biased than those derived from a single-level model. Additionally, results support the idea that polychoric ordinal a at level-1 of a two-level sampling design have slightly less bias across all data conditions than Cronbach’s a, and under normal and mixed data distributions for person reliability; however, the small gain in the precision of reliability estimates may not be worth the additional effort of calculating polychoric ordinal a for many clinicians and educators. Recommendations for Applied Researchers: Using Cronbach’s a under normal and mixed data conditions and across sample sizes is acceptable and easier to estimate due to its availability in social science software. For extremely non-normal data, the Rasch- RSM model should be used since the effort is worth the lower level of bias. The results also show that a variety of different data properties jointly affect reliability coefficients and care should be taken to provide both context and a theoretical framework in which to interpret results. Keywords: Reliability, Cronbach’s a, polychoric ordinal a, multilevel models, multilevel confirmatory factor analysis, Rasch item response theory, rating scale model

Extent

315 pages

Local Identifiers

Traxler_unco_0161D_10614

Rights Statement

Copyright is held by the author.

Download

Share

COinS