Original article | International Journal of Progressive Education 2018, Vol. 14(5) 144-154
The Effects of Q-Matrix Misspecification on Item and Model Fit
Seçil Ömür Sünbül & Semih Aşiret
pp. 144 - 154 | DOI: https://doi.org/10.29329/ijpe.2018.157.12 | Manu. Number: MANU-1808-30-0003
Published online: October 15, 2018 | Number of Views: 181 | Number of Download: 1027
Abstract
In this study it was aimed to evaluate the effects of various factors such as sample sizes, percentage of misfit items in the test and item quality (item discrimination) on item and model fit in case of misspecification of Q matrix. Data were generated in accordance with DINA model. Q matrix was specified for 4 attributes and 15 items. While data were generated, sample sizes as 1000, 2000, 4000, s and g parameters as low and high discrimination index were manipulated. Three different misspecified Q matrix (overspecified, underspecified and mixed) was developed considering the percentage of misfit items (%20 and %40). In the study, S-X2 was used as item fit statistics. Furthermore absolute (abs(fcor), max(X2)) and relative (-2 log-likelihood, Akaike's information criterion (AIC) and Bayesian information criterion (BIC)) model fit statistics were used. Investigating the results obtained from this simulation study, it was concluded that S-X2 was sufficient to detect misfit items. When the percentage of misfit items in the test was much or Q matrix was both underspecified and overspecified, the correct detection of both abs(fcor) and max(X2) statistics was approximately 1 or 1. In addition, the correct detection rates of both statistics was high under other conditions, too. AIC and BIC were successful to detect model misfit in the cases where the Q matrix underspecified, whereas, they were failed detect model misfit for other cases. It can be said that the performance of BIC was mostly better than other relative model fit statistics to detect model misfit.
Keywords: Cognitive diagnostic assessment, item fit, model fit, misspecification of Q matrix
| How to Cite this Article? |
|---|
|
APA 6th edition Harvard Chicago 16th edition |
| References |
|---|
|
Chen, J., de la Torre, J., & Zhang, Z. (2013). Relative and absolute fit evaluation in cognitive diagnosis modeling. Journal of Educational Measurement, 50, 123-140. Choi, H.-J., Templin, J. L., Cohen, A. S., & Atwood, C. H. (2010, April). The impact of model misspecification on estimation accuracy in diagnostic classification models. Paper presented at the meeting of the National Council on Measurement in Education (NCME), Denver, CO. De La Torre, J. (2009). A cognitive diagnosis model for cognitively-based multiple-choice options. Applied Psychological Measurement, 33, 163–183. De La Torre, J., & Lee, Y. S. (2013). Evaluating the Wald test for item-level comparison of saturated and reduced models in cognitive diagnosis. Journal of Educational Measurement, 50, 355-373. DiBello, L., Roussos, L. A., & Stout, W. F. (2007). Review of Cognitively Diagnostic Assessment and a Summary of Psychometric Models. In C. R. Rao & S. Sinharay (Eds.), Handbook of Statistics, 26, 979-1030. Galeshi & Skaggs (2014). Traditional fit indices utility in new psychometric model: cognitive diagnostic model. International Journal of Quantitative Research in Education, 2, 2. Kunina-Habenicht, O., Rupp, A. A., & Wilhelm, O. (2012). The impact of model misspecification on parameter estimation and item-fit assessment in log-linear diagnostic classification models. Journal of Educational Measurement, 49, 59-81. Lee, Y.W., & Sawaki, Y. (2009). Cognitive diagnostic approaches to language assessment: An overview. Language Assessment Quarterly, 6(3), 172-189. doi: 10.1080/15434300902985108 Li, H. (2016). Estimation of Q-matrix for DINA Model Using the Constrained Generalized DINA Framework. (doctoral dissertation). Coulumbia University. Liu, Y., Tian, W., & Xin, T. (2016). An application of M2 statistic to evaluate the fit of cognitive diagnostic models. Journal of Educational and Behavioral Statistics, 41, 3-26. Oliveri, M. E., & von Davier, M. (2011). Investigation of model fit and score scale comparability in international assessment. Psychological Test and Assessment Modeling, 53, 315-333. Orlando, M., & Thissen, D. (2000). Likelihood-based item-fit indices for dichotomous Item Response Theory models. Applied Psychological Measurement, 24(1), 50–64. Robitzsch, A., Kiefer, T., George, A. C., & Uenlue, A. (2015). CDM: Cognitive Diagnosis Modeling (R package Version 6.4-23). Retrieved from http://CRAN.R-project.org/package=CDM Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement,6(4), 219-262. Rupp, A. A., Henson, R. A., & Templin, J. L. (2010) Diagnostic measurement : theory, methods, and applications. Guilford Press Sinharay, S., & Almond, R. G. (2007). Assessing fit of cognitive diagnostic models: A case study. Educational and Psychological Measurement, 67, 239-257. Sorrel, Abad, Olea, de la Torre, & Barrada. (2017). Inferential Item-Fit Evaluation in Cognitive Diagnosis Modeling. Applied Psychological Measurement, 2017, 41, 614–631 Sorrel, M. A., Olea, J., Abad, F. J., de la Torre, J., Aguado, D., & Lievens, F. (2016). Validity and reliability of situational judgement test scores: A new approach based on cognitive diagnosis models. Organizational Research Methods, 19, 506-532. Tatsuoka, K. K. (1983). Rule space: An approach for dealing with misconceptions based on item response theory. Journal of Educational Measurement, 20, 345-354. Wang, C., Shu, Z., Shang, Z., & Xu, G. (2015). Assessing item level fit for the DINA model. Applied Psychological Measurement, 39, 525-538. |
