An application of cognitive diagnostic models in the evaluation of reading comprehension in Chilean university students
Una aplicación de los modelos de diagnóstico cognitivo en la evaluación de la comprensión lectora en universitarios chilenos
Aníbal Puente , Jesús M. Alvarado , Antonio Gutierrez de Blume , Juan Felipe Calderón Maureira
Suma Psicológica, (2023), 30(1), pp. 40-47.
Received 9 November 2022
Accept 21 April 2023
Introducción: Las habilidades de comprensión lectora son fundamentales para el rendimiento académico de los estudiantes universitarios. Estas habilidades no son innatas; por tanto, se requiere una formación sistemática antes de ingresar a la universidad. El objetivo principal de este artículo fue evaluar las habilidades de comprensión lectora de estudiantes universitarios chilenos. Método: En el estudio participó una muestra de 579 estudiantes que respondieron cuatro textos que componen las Estrategias de Comprensión Lectora para Estudiantes Universitarios (ELCU). Se utilizó una plataforma automatizada para aplicar y corregir la prueba con resultados y retroalimentación inmediatos. La herramienta permitió utilizar una muestra amplia y representativa de las distintas regiones de Chile. ELCU fue desarrollado mediante evaluación diagnóstica cognitiva (CDA) considerando estrategias cognitivas, metacognitivas y lingüísticas. Los análisis se realizaron mediante dos aproximaciones diferentes: The Linear Logistic Test Model (LLTM) y the Deterministic Inputs, Noisy “and” Gate (DINA) model. Resultados: El rechazo al modelo LLTM implica que las estrategias no difieren en complejidad cognitiva mientras que la adecuada bondad de ajuste al modelo DINA permiten caracterizar a los estudiantes según sus habilidades lectoras. Conclusión: La prueba ELCU permite obtener información confiable y válida sobre la capacidad de comprensión lectora y la conciencia metacognitiva de los estudiantes universitarios.
Palabras clave:
Teoría de respuesta al ítem, modelos de diagnóstico cognitivo, comprensión lectora
Introduction: Reading Comprehension skills are fundamental to the academic performance of university students. These abilities are not innate; therefore, systematic training is required before entering university. The main objective of this study was to evaluate the reading comprehension skills of Chilean University Students. Method: A sample of 579 students participated in the study, responding to four texts that comprise the Reading Comprehension Strategies for University Students (ELCU). An automated platform was used to apply and correct the test with immediate results and feedback. The tool allowed the use of a broad and representative sample of the various regions of Chile. The ELCU was developed using a cognitive diagnostic assessment (CDA) considering cognitive, metacognitive and linguistic strategies. The analyses were performed using two different approaches: The Linear Logistic Test Model (LLTM) and the deterministic inputs, noisy “and” gate (DINA) model. Results: The rejection of the LLTM implies that the strategies do not differ in cognitive complexity (difficulty) while the adequate goodness of fit to the DINA model allows to characterise the students according to their reading abilities. Conclusion: The ELCU test provides reliable and valid information on the reading comprehension ability and metacognitive awareness of university students.
Keywords:
Item response theory, cognitive diagnostic models, reading comprehension
Abu-Snoubar, T. K. (2017). An evaluation of EFL students’ attitudes toward English language learning in terms of several variables. International Journal of English Language Teaching, 5(6), 18-34.
Afflerbach, P., Person, D., & Paris, S. (2008). Clarifying the differences between reading skills and reading strategies The Reading Teacher, 61(5), 364-373. https://doi.org/10.1598/RT.61.5.1
Aktar, T., & Ahmed, I. (2018). Metacognitive awareness of strategy use in EFL reading comprehension in Bangladesh. Journal of ELT Research: The Academic Journal of Studies in English Language Teaching and Learning, 3(2), 204-221. https://doi.org/10.22236/JER_Vol3Issue2pp204-221
Alami, M. (2016). Cross-gender comparison of metacognitive strategies utilized by omani students in reading comprehension classes. International Journal of Applied Linguistics & English Literature, 5(4), 20-28. https://doi.org/10.7575/aiac.ijalel.v.5n.4p.20
Ali, A., & Razali, A. (2019). A review of studies on cognitive and metacognitive reading strategies in teaching reading comprehension for ESL/EFL learners. English Language Teaching, 12(6), 94-110. https://doi.org/10.5539/elt.v12n6p94
Almanza, E. C., Montoya, J. A. G., Montero, J. S. N., Quispe, G. V., & Fernández, D. M. M. (2022). Estrategias metacognitivas en la comprensión lectora en estudiantes de la educación básica regular. Revista Conrado, 18(88), 308-317.
Alvarado, J. M., Puente, A., Fernández, M. P., & Jiménez, V. (2015). Análisis de los componentes en la adquisición de la lectura en castellano: una aplicación del modelo logístico lineal. Suma Psicológica, 22(1), 45-52. https://doi.org/10.1016/j.sumpsi.2015.05.006
Andersen, E. B. (1973). A goodness of fit test for the Rasch model. Psychometrika, 38, 123–140. https://doi.org/10.1007/BF02291180
Barreyro, J. P., Injoque-Ricle, I., Álvarez-Drexler, A., Formoso, J., & Burin, D. I. (2017). Generación de inferencias explicativas en la comprensión de textos expositivos: el rol de la memoria de trabajo y el conocimiento previo específico. Suma Psicológica, 24(1), 17-24. https://doi.org/10.1016/j.sumpsi.2016.09.002
Bashir, I., & Mattoo, N. H. (2012). A study-on-study habits and academic performance among adolescents (14-19) years. International Journal of Social Science Tomorrow, 1(5), 1-5.
Berrocal, M., & Ramírez, F. (2019). Estrategias metacognitivas para desarrollar la comprensión lectora. Revista Innova Educación, 1(4), 522-545. https://doi.org/10.35622/j.rie.2019.04.008
Cáceres-Serrano, P., & Alvarado-Izquierdo, J. M. (2019). Evaluación de la comprensión lectora mediante un modelo de diagnóstico cognitivo. Revista Iberoamericana de Diagnóstico y Evaluación-e Avaliação Psicológica, 2(51), 149-162. https://doi.org/10.21865/RIDEP51.2.11
Calderón, J. F., Puente, A., & Mendoza, M. (2020). Academic literacy in engineering students: Formulation of an instrument to assess reading comprehension and metacognitive strategies. INTED, EduLearn20 Proceedings.
Carlino, P. (2009). Leer y escribir en la universidad, una nueva cultura. ¿Por qué es necesaria la alfabetización académica? Página y Signos, 3(5), 13-52. https://www.aacademica.org/paula.carlino/85
Castrillón Rivera, E. M., Morillo Puente, S., & Restrepo Calderón, L. A. (2020). Diseño y aplicación de estrategias metacognitivas para mejorar la comprensión lectora en estudiantes de secundaria. Ciencias Sociales y Educación, 9(17), 203-231. https://doi.org/10.22395/csye.v9n17a10
Cimmiyotti, C.B. (2013). Impact of reading ability on academic performance at the primary level. Master’s theses and capstone projects (Paper 127). https://doi.org/10.33015/dominican.edu/2013.edu.18
Dardjito, H. (2019). Students’ metacognitive reading awareness and academic english reading comprehension in EFL context. International Journal of Instruction, 12(4), 611-624. https://doi.org/10.29333/iji.2019.12439a
De La Torre, J., & Douglas, J. A. (2004). Higher-order latent trait models for cognitive diagnosis. Psychometrika, 69(3), 333-353. https://doi.org/10.1007/BF02295640
Ferrara, A. M., & Panlilio, C. C. (2020). The role of metacognition in explaining the relationship between early adversity and reading comprehension. Children and Youth Services Review, 112,104884. https://doi.org/10.1016/j.childyouth.2020.104884
George, A. C., Robitzsch, A., Kiefer, T., Groß, J., & Ünlü, A. (2016). The R package CDM for cognitive diagnosis models. Journal of Statistical Software, 74(2), 1-24. https://doi.org/10.18637/jss.v074.i02
González, M., & Romero, J. (2000). Las dificultades en comprensión lectora. En J. García-Sánchez (coord.), De la Psicología de la instrucción a las necesidades curriculares. Oikos-Tau. 29-239.
Graesser, A. C. (2007). An introduction to strategic reading comprehension. In D. S. McNamara (Ed.), Reading comprehension strategies: Theories, interventions, and technologies (pp. 3-26). Erlbaum.
Gutierrez, A. P., & Schraw, G. (2015). Effects of strategy training and incentives on students’ performance, confidence, and calibration. Journal of Experimental Education, 83, 386-404. https://doi.org/10.1080/00220973.2014.907230
Ma, W., & de la Torre, J. (2018). GDINA: The generalized DINA model framework. R package version 2.1.
Mair, P., & Hatzinger, R. (2007). Extended Rasch modeling: The eRm package for the application of IRT models in R. Journal of Statistical Software, 20(9), 1-20. https://doi.org/10.18637/jss.v020.i09
McNamara, D. (2004). Aprender del texto: efectos de la estructura textual y las estrategias del lector. Revista Signos, 37(55), 1-12. https://doi.org/10.4067/S0718-09342004005500002
McNamara, D. (2007). Reading comprehension strategies: Theories, interventions, and technologies. Routlegge/Taylos & Francis.
Miyamoto, A., Pfost, M., & Artelt, C. (2019). The relationship between intrinsic motivation and reading comprehension: Mediating effects of reading amount and metacognitive knowledge of strategy use. Scientific Studies of Reading, 23(6), 445-460. https://doi.org/10.1080/10888438.2019.1602836
Mokhtari, K., Dimitrov, D. M., & Reichard, C. A. (2018). Revising the metacognitive awareness of reading strategies inventory (marsi) and testing for factorial invariance. Studies in Second Language Learning and Teaching, 8(2), 219-246. https://doi.org/10.14746/ssllt.2018.8.2.3
Mokhtari, K., & Reichard, C. A. (2002). Assessing students’ metacognitive awareness of reading strategies. Journal of Educational Psychology, 94(2), 249-259. https://doi.org/10.1037/0022-0663.94.2.249
Murphy, P. K., Wilkinson, I. A. G., Soter, A. O., Hennessey, M. N., & Alexander, J. F. (2009). Examining the effects of classroom discussion on students’ comprehension of text: A meta-analysis. Journal of Educational Psychology, 101(3), 740-764. https://doi.org/10.1037/a0015576
Narvaja, E., Di Stefano, M., & Pereira, C. (2003). La lectura y la escritura en la universidad. Eudeba.
Oliveira, K. L., & Santos, A. A. A. (2006). Compreensão de textos e desempenho acadêmico. Psic – Revista de Psicologia da Vetor Editora, 7(1), 19-27.
Puente, A., Mendoza-Lira, M., Calderón, J. F., & Zúñiga, C. (2019). Estrategias metacognitivas lectoras para construir el significado y la representación de los textos escritos. Ocnos. Revista de Estudios sobre Lectura, 18(1), 21-30. https://doi.org/10.18239/ocnos_2019.18.1.1781
Rabia, M., Mubarak, N., Tallat, H., & Nasir, W. (2017). A study on study habits and academic performance of students. International Journal of Asian Social Science, 7(10), 891-897. https://doi.org/10.18488/journal.1.2017.710.891.897
Rastegar, M., Kermani, E. M., & Khabir, M. (2017). The relationship between metacognitive reading strategies and reading comprehension among 2nd year EFL university students. https://doi.org/10.4236/ojml.2017.72006
Rupp, A. A., & Templin, J. L. (2008). Unique characteristics of diagnostic classification models: A comprehensive review of the current state-of-the-art. Measurement, 6(4), 219-262. https://doi.org/10.1080/15366360802490866
Schraw, G. (2009). A conceptual analysis of five measures of metacognitive monitoring. Metacognition and Learning, 4(1), 33-45. https://doi.org/10.1007/s11409-008-9031-3
Schraw, G., & Denninson, R. S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology, 19, 460-475. https://doi.org/10.1006/ceps.1994.1033
Sheikh, I., Soomro, K. A., & Hussain, N. (2019). Metacognitive awareness of reading strategies, reading practices and academic attainments of university students. Journal of Education and Educational Development, 6(1), 126-137. https://eric.ed.gov/?id=EJ1216772
Soto, C., Gutierrez de Blume, A.P., Asun, R., Jacovina, M., & Vásquez, V. (2018). A deeper understanding of metacomprehension: Development of new multidimensional tool. Frontline Learning Research, 6(1), 31-52. https://doi.org/10.14786/flr.v6i1.328
Soto, C., Gutiérrez de Blume, A. P., Jacovina, M., McNamara, D., Benson, N., & Riffo, B. (2019). Reading comprehension and metacognition: The importance of inferential skills. Cogent Education, 6(1), 1565067. https://doi.org/10.1080/2331186X.2019.1565067
Teng, F. (2020). The benefits of metacognitive reading strategy awareness instruction for young learners of English as a second language. Literacy, 54(1), 29-39. https://doi.org/10.1111/lit.12181
Vega, N. A., Bañales, G., & Correa, S. (2011). ¿Cómo los estudiantes universitarios autorregulan su comprensión cuando leen múltiples textos científicos? XI Congreso Nacional de Investigación Educativa, Ciudad de México.
Wright, B. D. (1977). Solving measurement problems with the Rasch model. Journal of Educational Measurement, 97-116. https://doi.org/10.1111/j.1745-3984.1977.tb00031.x
Wudeneh, A. M. (2018). The relationship between metacognitive reading strategy awareness and reading comprehension among freshman EFL students, Ethiopia. The Journal of English Language and Literature, 10, 1043-1050.
