Keywords
assessment and evaluation
middle school students
learning gaps
solar system and beyond
How to Cite
Abstract
This study investigates the impact of ChatGPT-supported assessments on addressing students’ learning gaps. With the increasing integration of artificial intelligence (AI) in education, understanding its potential benefits in enhancing students’ academic achievement is of great importance. For this purpose, a quasi-experimental design was implemented within a quantitative methodology framework. The study involved seventh-grade students from a public middle school in Turkey during the 2024-2025 academic year. To assess students’ learning levels, seventh-grade assessment questions from the “The Solar System and Beyond” unit, published by the Ministry of National Education (MEB), were used. These multiple-choice questions consisted of four options, with one correct answer, and underwent validity and reliability testing. Prior to publication, the Ministry of National Education (MoNE) sought expert opinions and conducted pilot implementations to analyze item performance. All questions were designed to be appropriate for students’ developmental levels and were explicitly aligned with the "Solar System and Beyond" unit.The experimental group participated in a ChatGPT-supported question-and-answer process during the “The Solar System and Beyond” unit. Instead of providing direct answers, students received step-by-step guidance toward the correct solution and immediate feedback. The results showed that ChatGPT-supported assessments effectively addressed learning gaps, and the experimental group demonstrated statistically significant improvement in learning outcomes. This study provides valuable insights into the potential of ChatGPT-supported chatbots to reduce learning gaps in educational settings and suggests that they can contribute to improving students’ academic performance. Additionally, the study offers new empirical evidence that AI-supported chatbots can actively facilitate learning by supporting students’ reasoning processes rather than merely providing correct answers, thus presenting a novel approach to AI-supported formative assessment in education.
References
[1] N. P. Melville, L. Robert, and X. Xiao, "Putting humans back in the loop: An affordance conceptualization of the 4th industrial revolution," Information Systems Journal, vol. 33, no. 4, pp. 733–757, 2023. [Online]. Available: https://doi.org/10.1111/isj.12422 (in English).
[2] K. D. Caratiquit and L. J. C. Caratiquit, "ChatGPT as an academic support tool on the academic performance among students: The mediating role of learning motivation," Journal of Social, Humanity, and Education, vol. 4, no. 1, pp. 21–33, 2023. [Online]. Available: https://doi.org/10.35912/jshe.v4i1.1558 (in English).
[3] S. Wolfram, What is ChatGPT doing … and why does it work?, Wolfram Media Inc., 2023. [Online]. Available: https://doi.org/10.31855/bc47ee6b-75c (in English).
[4] Y. Su, Y. Lin, and C. Lai, "Collaborating with ChatGPT in argumentative writing classrooms," Assessing Writing, vol. 57, p. 100752, 2023. [Online]. Available: https://doi.org/10.1016/j.asw.2023.100752 (in English).
[5] R. H. Mogavi et al., "Exploring user perspectives on ChatGPT: Applications, perceptions, and implications for AI-integrated education," 2023. [Online]. Available: https://doi.org/10.48550/ARXIV.2305.13114 (in English).
[6] J. A. Flad, The effects on increasing cognitive load on self-report and dual task measures of mental effort during problem solving, Doctoral dissertation, University of Southern California, 2002. (in English).
[7] U. A. Bukar et al., "Text analysis on early reactions to ChatGPT as a tool for academic progress or exploitation," SN Computer Science, vol. 5, p. 366, 2024. [Online]. Available: https://doi.org/10.1007/s42979-024-02714-7 (in English).
[8] M. Sullivan, A. Kelly, and P. McLaughlan, "ChatGPT in higher education: Considerations for academic integrity and student learning," Journal of Applied Learning & Teaching, vol. 6, no. 1, pp. 1–10, 2023. [Online]. Available: http://dx.doi.org/10.37074/jalt.2023.6.1.17 (in English).
[9] J. Willems, "ChatGPT at universities - the least of our concerns," 2023. [Online]. Available: https://doi.org/10.2139/ssrn.4334162 (in English).
[10] J. Skalka, M. Drlík, and J. Obonya, "Automated assessment in learning and teaching programming languages using virtual learning environment," in 2019 IEEE Global Engineering Education Conference (EDUCON), 2019. [Online]. Available: https://doi.org/10.1109/EDUCON.2019.8725127 (in English).
[11] J. Steiss et al., "Comparing the quality of human and ChatGPT feedback of students’ writing," Learning and Instruction, vol. 91, 2024. [Online]. Available: https://doi.org/10.1016/j.learninstruc.2024.101790 (in English).
[12] A. P. Cavalcanti et al., "Automatic feedback in online learning environments: A systematic literature review," Computers and Education: Artificial Intelligence, vol. 2, p. 100027, 2021. [Online]. Available: https://doi.org/10.1016/j.caeai.2021.100027 (in English).
[13] W. Dai et al., "Can large language models provide feedback to students? A case study on ChatGPT," in 23rd IEEE International Conference on Advanced Learning Technologies (ICALT), 2023. [Online]. Available: https://doi.org/10.1109/ICALT58122.2023.00100 (in English).
[14] L. Wang et al., "ChatGPT's capabilities in providing feedback on undergraduate students’ argumentation: A case study," Thinking Skills and Creativity, vol. 51, p. 101440, 2024. [Online]. Available: https://doi.org/10.1016/j.tsc.2023.101440 (in English).
[15] A. C. F. Tam, "Interacting with ChatGPT for internal feedback and factors affecting feedback quality," Assessment and Evaluation in Higher Education, pp. 1–17, 2024. [Online]. Available: https://doi.org/10.1080/02602938.2024.2374485 (in English).
[16] S. Alneyadi and Y. Wardat, "ChatGPT: Revolutionizing student achievement in the electronic magnetism unit for eleventh-grade students in Emirates schools," Contemporary Educational Technology, vol. 15, no. 4, p. 448, 2023. [Online]. Available: https://doi.org/10.30935/cedtech/13417 (in English).
[17] J. Hattie and H. Timperley, "The power of feedback," Review of Educational Research, vol. 77, no. 1, pp. 81–112, 2007. [Online]. Available: https://doi.org/10.3102/003465430298487 (in English).
[18] X. Yang, Q. Wang, and J. Lyu, "Assessing ChatGPT’s educational capabilities and application potential," ECNU Review of Education, 2023. [Online]. Available: https://doi.org/10.1177/20965311231210006 (in English).
[19] J. W. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches, 4th ed., Sage Publications, 2014. (in English).
[20] J. Cohen, Statistical power analysis for the behavioral sciences, 2nd ed., Erlbaum, 1988. (in English).
[21] B. G. Tabachnick and L. S. Fidell, Using multivariate statistics, 7th ed., Pearson, 2019. (in English).
[22] J. R. Fraenkel, N. E. Wallen, and H. H. Hyun, How to design and evaluate research in education, 8th ed., McGraw-Hill, 2012. (in English).
[23] O. Zawacki-Richter, V. I. Marín, M. Bond, and F. Gouverneur, "Systematic review of research on artificial intelligence applications in higher education – where are the educators?" International Journal of Educational Technology in Higher Education, vol. 16, no. 1, p. 39, 2019. [Online]. Available: https://doi.org/10.1186/s41239-019-0171-0 (in English).
[24] S. Knight, A. F. Wise, and B. Chen, "The impact of using artificial intelligence in classrooms on student learning outcomes," Journal of Educational Computing Research, vol. 58, no. 4, pp. 780–804, 2020. (in English).
[25] V. J. Shute, E. G. Hansen, and R. G. Almond, "You can't fatten a hog by weighing it—or can you? Evaluating an assessment for learning system called ACED," International Journal of Artificial Intelligence in Education, vol. 18, no. 4, pp. 289–316, 2008. (in English).
[26] K. Arslan, "Artificial intelligence and its applications in education," Western Anatolia Journal of Educational Sciences, vol. 11, no. 1, pp. 71–88, 2020. (in English).
[27] C. Kooli, "Chatbots in education and research: A critical examination of ethical implications and solutions," Sustainability, vol. 15, no. 7, p. 5614, 2023. [Online]. Available: https://doi.org/10.3390/su15075614 (in English).
[28] P. Bruneau, J. Wang, L. Cao, and H. Trương, "The potential of ChatGPT to enhance physics education in Vietnamese high schools," 2023. [Online]. Available: https://doi.org/10.35542/osf.io/36qw9 (in English).
[29] K. Taneja et al., "Jill Watson: A virtual teaching assistant powered by ChatGPT," in International Conference on Artificial Intelligence in Education, pp. 324–337, Springer Nature Switzerland, 2024. [Online]. Available: https://doi.org/10.1007/978-3-031-64302-6_23 (in English).
[30] I. Kostka and R. Toncelli, "Exploring applications of ChatGPT to English language teaching: Opportunities, challenges, and recommendations," TESL-EJ, vol. 27, no. 3, 2023. [Online]. Available: https://doi.org/10.55593/ej.27107int (in English).
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright (c) 2025 Mehmet Akif Arduç, Gülşah Gürkan