Keywords
criterion-oriented test
analyses of test items
How to Cite
Abstract
One of the topical current issues of higher education is issue of monitoring students' knowledge in the distance learning mode, in particular, conducting semester exams. The objective of the article is analyze of the methods and means of exam control in test form in the conditions of distance learning and assessing the reliability of the test results. We formulated the requirements to the tasks of examination test, considered technology of evaluation of statistical parameters of quality of tasks using built-in LMS Moodle mathematical package and free software application packages ltm, eRm, mirt of the CRAN library of R programming environment. Experience has shown that the use of these software tools is an optimal and effective way to control the quality of test tasks within the framework of classical and modern test theories. We propose to use tests of three levels, where the difficulty of each item is determined by the results of statistical analysis. Consistent performance of such tasks allows you to establish the level of mastery of knowledge by each student. It is shown that individualization of test tasks, that execute students, creations of variety their form, in particular, the use of multiple-choice tasks with several correct answers, open-type tasks with a numerical answer, plugging of graphic data in a task substantially complicates the probable students’ cheating during the exam. We substantiated the possibility of using the LMS Moodle together with the Google Meet Pro or Zoom video conference program to conduct the exam without using special proctoring programs. It is emphasized that the verbal teacher-student communication allows not only to draw a conclusions about the independence of solving the tasks, but also assists to the development of students' scientific communication skills.
The results of the calculation of generalization coefficients, Cohen kappa and correlation phi-coefficient made it possible to conclude about the reliability of assessing the educational achievements of students based on the results of their performance of tasks of a criterion-oriented exam test.
References
М. С. Паккі, “Забезпечення надійності процедур оцінювання здобувачів в умовах онлайн навчання.” ITLT, том 88, вип. 2, с. 139–151, Квіт 2022. DOI: https://doi.org/10.33407/itlt.v88i2.4476
В. Ю. Биков, Ю. М. Богачков, Ю. О. Жук. Моніторинг рівня навчальних досягнень з використанням Інтернет технологій. Київ, Україна: Педагогічна думка, 2008.
В. П. Сергієнко, Л. О. Кухар. Методичні рекомендації зі складання тестових завдань : Київ, Україна : Вид-во НПУ, 2011.
Winsteps. [Електронний ресурс]. Доступно: https://www.winsteps.com/winsteps.htm. Дата звернення: лип. 12, 2022.
The Comprehensive R Archive Network. [Електронний ресурс]. Доступно: https://cran.r-project.org/. Дата звернення: лип. 22, 2022.
М. С. Мазорчук, В. С. Добряк, П. С. Емельянов, “Методы и модели анализа качества тестовых заданий и моделирование компьютерного адаптивного тестирования в системах дистанционного обучения,” Открытые информационные и компьютерные интегрированные технологии. - Вып. 73, c. 103-117, 2016. [Електронний ресурс]. Доступно: http://nbuv.gov.ua/UJRN/vikt_2016_73_14. Дата звернення: лип. 22, 2022.
S. Castro, “Psychometrics”. [Електронний ресурс]. Доступно: https://rstudio-pubs-static.s3.amazonaws.com/141954_4774223853064561a65bf2f9dc2256ae.html. Дата звернення: лип. 02, 2022.
F. Noorbehbahani, A. Mohammadi, M. Aminazadeh, “A systematic review of research on cheating in online exams from 2010 to 2021”, Educ Inf Technol, 2022. Доступно: https://doi.org/10.1007/s10639-022-10927-7. Дата звернення: лип. 15, 2022.
O. L. Holden, M. E. Norris, V. A. Kuhlmeier, “Academic Integrity in Online Assessment,” : A Research Review. Frontiers in Education, vol.6. 2021. doi: 10.3389/feduc.2021.639814. Доступно: https://www.frontiersin.org/articles/10.3389/feduc.2021.639814/full. Дата звернення: лип. 08, 2022.
L. L. Walsh, D. A. Lichti, C. M. Zambrano-Varghese, A. D. Borgaonkar, J. S. Sodhi, S. Moon, E. R. Wester and, K. L. Callis-Duehl, “Why and how science students in the United States think their peers cheat more frequently online: perspectives during the COVID-19 pandemic” International Journal for Educational Integrity, vol.17, no.23, 2021. Доступно: https://doi.org/10.1007/s40979-021-00089-3.
M. W. Romaniuk, J. Łukasiewicz-Wieleba, “Academic Lecturers Towards the Students' Examining. Similarities and Differences of Stationary and Remote Exams in the Pandemic Era” International Journal of Electronics and Telecommunications, vol. 68, no.1, pp. 63-68, 2022, doi: 10.24425/ijet.2022.139849.
E. Broemer, G. Recktenwald, “Cheating and Chegg: A Retrospective” presented at 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference, July 2021, doi: 10.18260/1-2—36792. Доступно: https://peer.asee.org/cheating-and-chegg-a-retrospective. Дата звернення: лип. 23, 2022.
J. Ranger, N. Schmidt, A. Wolgast, “The Detection of Cheating on E-Exams in Higher Education—The Performance of Several Old and Some New Indicators” (2020) Frontiers in Psychology, 11, art. no. 568825, doi: 10.3389/fpsyg.2020.568825. Доступно: https://www.frontiersin.org/articles/10.3389/fpsyg.2020.568825/full. Дата звернення: лип. 23, 2022.
N. Thompson, How To Pass An Online Proctored Exam. [Електронний ресурс]. Доступно: https://assess.com/pass-online-proctored-exam/ Дата звернення: черв. 20, 2022.
How to set up your room for a remote proctored exam through MonitorEDU. [Електронний ресурс]. Доступно: https://www.youtube.com/watch?v=cInMjEaH1q0 Дата звернення: черв. 30, 2022.
1000 и 1 способ обойти Safe Exam Browser, [Електронний ресурс]. Доступно: https://habr.com/ru/company/neobit/blog/512678/ Дата звернення: лип. 18, 2022.
О. О. Романовський, Ю. Ю. Романовська, Н. М. Чаплинська, С. А. Данилов, “Онлайн контроль самостійної роботи студентів в умовах дистанційного навчання,” Інформаційні технології і засоби навчання, т. 84, №4, 211–233, 2021. https://doi.org/10.33407/itlt.v84i4.4339
Z. Jiang and J. Huang, "Effective and Efficient Strategies and Their Technological Implementations to Reduce Plagiarism and Collusions in Non-proctored Online Exams," in IEEE Transactions on Learning Technologies, 1 Feb. 2022, pp. 107-118 doi: 10.1109/TLT.2022.3153948.
K. Linden, P. Gonzalez, “Zoom invigilated exams: A protocol for rapid adoption to remote examinations” (2021) British Journal of Educational Technology, vol. 52, no.4, pp. 1323 - 1337, 2021, doi: 10.1111/bjet.13109
N. P. Kalashnikov, S. S. Muravyev-Smirnov, D. A. Samarchenko, A. N. Tyulyusov, “Remote Access Application for General Physics Examinations in the Magistracy Institute of National Research Nuclear University MEPhI,” AIP Conference Proceedings, vol.1797, issue 1, 2017. Доступно: https://doi.org/10.1063/1.4972425. Дата звернення: лип. 15, 2022.
Міністерство освіти і науки України (2020). Система та загальні критерії оцінювання результатів навчання учнів. [Електронний ресурс]. Доступно: https://mon.gov.ua/storage/app/media/regulatorna_dijalnist/2020/09/14/Systema%20otsinyuvannya/Proyekt%20nakazu%20MON%20Systema%20otsinyuvannya.pdf.
L. Crocker, J. Algina, Introduction to Classical and Modern Test Theory, Belmont, CA: Wadsworth, 527 p., 2006.
L. Bilousova, O. Kolgatin, L. Kolgatina, "Pedagogical Diagnostics with Use of Computer Technologies", CEUR Workshop Proceedings, vol. 1000, pp. 209–220. [Електронний ресурс]. Доступно: http://ceur-ws.org/Vol-1000/ICTERI-2013-p-209-220.pdf
А. Н. Майоров, Теория и практика создания тестов для системы образования. (Как выбирать, создавать и использовать тесты для целей образования). Москва, Россия, «Интеллект-центр», 2001.
Ю. Жук, Л. Ващенко, “Вплив форми тестових завдань і профільної спеціалізації класів на результати виконання здобувачами середньої освіти тесту з біології,” Український педагогічний журнал, №3, с.90-99, 2019 .
С. О. Подласов, “Тестування з фізики,” Вісник Чернігівського державного педагогічного університету ім. Т. Г. Шевченка, серія: педагогічні науки, вип. 23, с.107-111, 2004.
В. М. Кухаренко, Л. П. Перхун, Н. М. Товмаченко, “Методика комплексного оцінювання якості тестів. Частина 2,” Статистика України, №4, с.72-79, 2018.
V. P. Serhiienko, L. О. Kukhar, O. V. Halytskyi, & P. V. Mykytenko. “Use of integrated system of analysis of tests’ tasks in LCMS MOODLE”. Information Technologies and Learning Tools, vol. 41, no. 3, May 2014, pp. 196-08, doi:10.33407/itlt.v41i3.979.
W. Angoff, “Scales. Norms. And equivalent scores,” in Educational Measurement. R. L. Thorndike (Ed.), – Washington, DC.: American Council on Education. 1971, pp. 508 – 600
Что такое система прокторинга и как её обойти. [Електронний ресурс]. Доступно: https://zaochnik.ru/blog/chto-takoe-sistema-proktoringa-i-kak-ee-obojti. Дата звернення: лип. 23, 2022.
REFERENCES (TRANSLATED AND TRANSLITERATED)
M. S. Pakki, “Тhe ensuring of reliability of students’ assessment procedures during online learning”. ITLT, vol.88, no. 2, pp.139-151, apr. 2022. doi: https://doi.org/10.33407/itlt.v88i2.4476. (in Ukrainian).
V. Yu. Bykov, Yu. M. Bohachkov, Yu. O. Zhuk, Monitoring the level of educational achievements using Internet technologies. Kyiv, Ukraina, ed. Pedahohichna dumka, 2008. (in Ukrainian).
V. P. Serhiyenko, L. O. Kukhar, Methodical recommendations for writing test items, Kyiv, Ukraina, ed. NPU, 2011. (in Ukrainian).
Winsteps. [Online]. Available: https://www.winsteps.com/winsteps.htm. Accessed on: July 12, 2022. (in English)
The Comprehensive R Archive Network. [Online]. Available: https://cran.r-project.org/. Accessed on: July 22, 2022. (in English)
M. S. Mazorchuk, V. S. Dobryak, P. S. Emel'yanov, “ Methods and models of test task quality analysis and modeling of computer adaptive testing in distance learning systems,” Open information and computer integrated technologies, nb.73, pp. 103-117, 2016. [Online]. Available: http://nbuv.gov.ua/UJRN/vikt_2016_73_14. Accessed on: July 22, 2022. (in Russian).
S. Castro, “Psychometrics”. [Online]. Available: https://rstudio-pubs-static.s3.amazonaws.com/141954_4774223853064561a65bf2f9dc2256ae.html. Accessed on: July 02, 2022. (in English).
F. Noorbehbahani, A. Mohammadi, M. Aminazadeh, “A systematic review of research on cheating in online exams from 2010 to 2021”, Educ Inf Technol, 2022. Available: https://doi.org/10.1007/s10639-022-10927-7. Accessed on: July. 15, 2022. (in English).
O. L. Holden, M. E. Norris, V. A. Kuhlmeier, “Academic Integrity in Online Assessment,” : A Research Review. Frontiers in Education, vol.6. 2021. doi: 10.3389/feduc.2021.639814. Available: https://www.frontiersin.org/articles/10.3389/feduc.2021.639814/full. Accessed on: July. 08, 2022. (in English).
L. L. Walsh, D. A. Lichti, C. M. Zambrano-Varghese, A. D. Borgaonkar, J. S. Sodhi, S. Moon, E. R. Wester and, K. L. Callis-Duehl, “Why and how science students in the United States think their peers cheat more frequently online: perspectives during the COVID-19 pandemic” International Journal for Educational Integrity, vol.17, no.23, 2021. doi: https://doi.org/10.1007/s40979-021-00089-3. (in English)
M. W. Romaniuk, J. Łukasiewicz-Wieleba, “Academic Lecturers Towards the Students' Examining. Similarities and Differences of Stationary and Remote Exams in the Pandemic Era,” International Journal of Electronics and Telecommunications, vol. 68, no.1, pp. 63-68, 2022, doi: 10.24425/ijet.2022.139849. (in English).
E. Broemer, G. Recktenwald, “Cheating and Chegg: A Retrospective”, presented at 2021 ASEE Virtual Annual Conference Content Access, Virtual Conference, July 2021, doi: 10.18260/1-2—36792. Available: https://peer.asee.org/cheating-and-chegg-a-retrospective. Accessed on: July 23, 2022. (in English).
J. Ranger, N. Schmidt, A. Wolgast, “The Detection of Cheating on E-Exams in Higher Education—The Performance of Several Old and Some New Indicators” (2020) Frontiers in Psychology, 11, art. no. 568825, doi: 10.3389/fpsyg.2020.568825. Available: https://www.frontiersin.org/articles/10.3389/fpsyg.2020.568825/full, Accessed on: July 23, 2022. (in English).
N. Thompson, How To Pass An Online Proctored Exam. [Online]. Available: https://assess.com/pass-online-proctored-exam/. Accessed on: July. 20, 2022. (in English)
How to set up your room for a remote proctored exam through MonitorEDU. [Online]., Available: https://www.youtube.com/watch?v=cInMjEaH1q0 Accessed on: June. 30, 2022. (in English).
1000 & 1 way around Safe Exam Browser. [Online]. Available: https://habr.com/ru/company/neobit/blog/512678/ Accessed on: June 18, 2022. (in Russian)
O. O. Romanovskyi, Yu. Yu. Romanovska, N. M. Chaplynska, S. A. Danylov, “Online control of students’ independent work in the conditions of distance learning,”Information Technologies and Learning Tools, vol. 84, no.4, pp.211–233, (in Ukrainian), 2021. doi: https://doi.org/10.33407/itlt.v84i4.4339 (in Ukrainian).
Z. Jiang and J. Huang, "Effective and Efficient Strategies and Their Technological Implementations to Reduce Plagiarism and Collusions in Non-proctored Online Exams," in IEEE Transactions on Learning Technologies, 1 Feb. 2022, pp. 107-118 doi: 10.1109/TLT.2022.3153948. (in English).
K. Linden, P. Gonzalez, “Zoom invigilated exams: A protocol for rapid adoption to remote examinations” (2021) British Journal of Educational Technology, vol. 52, no.4, pp. 1323 - 1337, 2021, doi: 10.1111/bjet.13109 (in English).
N. P. Kalashnikov, S. S. Muravyev-Smirnov, D. A. Samarchenko, A. N. Tyulyusov, “Remote Access Application for General Physics Examinations in the Magistracy Institute of National Research Nuclear University MEPhI,” AIP Conference Proceedings, vol.1797, issue 1, 2017.doi: https://doi.org/10.1063/1.4972425. Accessed on: June 15, 2022. (in English).
Ministry of Education and Science of Ukraine (2020). The system and general criteria for evaluating student learning outcomes. [Online]. Available: https://mon.gov.ua/storage/app/media/regulatorna_dijalnist/2020/09/14/Systema%20otsinyuvannya/Proyekt%20nakazu%20MON%20Systema%20otsinyuvannya.pdf. Accessed on: October 15, 2022. (in Ukrainian)
L. Crocker, J. Algina, Introduction to Classical and Modern Test Theory, Belmont, CA: Wadsworth, 527 p., 2006. (in English).
L. Bilousova, O. Kolgatin, L. Kolgatina, "Pedagogical Diagnostics with Use of Computer Technologies", CEUR Workshop Proceedings, vol. 1000, pp. 209–220. [Online]. Available: http://ceur-ws.org/Vol-1000/ICTERI-2013-p-209-220.pdf. Accessed on: October 12, 2022. (in English)
A. N. Mayorov, Theory and practice of creating tests for the education system. (How to select, create and use tests for educational purposes). Moscow, Russia, “Intellekt-tsentr”, 2001. (in Russian).
Yu. Zhuk, L. Vashchenko, “ The influence of the form of test tasks and the profile specialization of classes on the results of secondary education students' performance of the biology test,” Ukrainian Pedagogical Journal, no.3, pp.90-99, (in Ukrainian), 2019.
S. O. Podlasov, “ Physics testing,” Bulletin of the Chernihiv State Pedagogical University named after T. G. Shevchenko, series: pedagogical sciences, vol. 23, pp.107-111, 2004. (in Ukrainian).
V. M. Kukharenko, L. P. Perkhun, N. M. Tovmachenko, “ Methodology of complex assessment of the quality of tests. Part 2,” Statistics of Ukraine, no. 4, pp.72-79, 2018. (in Ukrainian).
V. P. Serhiienko, L. О. Kukhar, O. V. Halytskyi, & P. V. Mykytenko. “Use of integrated system of analysis of tests’ tasks in LCMS MOODLE”. Information Technologies and Learning Tools, vol. 41, no. 3, May 2014, pp. 196-08, doi:10.33407/itlt.v41i3.979. (in Ukrainian).
W. Angoff, “Scales. Norms. And equivalent scores,” in Educational Measurement. R. L. Thorndike (Ed.), – Washington, DC.: American Council on Education. 1971, pp. 508 – 600. (in English)
What is the proctoring system and how to bypass it. [Online]. Available: https://zaochnik.ru/blog/chto-takoe-sistema-proktoringa-i-kak-ee-obojti, Accessed on: July. 23, 2022. (in Russian)
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright (c) 2022 Сергій Олександрович Подласов, Олексій Васильович Матвійчук