Keywords
СhatGPT
testing and assessment
foreign language competence
survey
How to Cite
Abstract
The paper presents the results of the first stage of the research into the specifics of ChatGPT application in assessment development for English as a Foreign Language (EFL) courses. The article aims to investigate the perceptions of English language teachers regarding the utilization of ChatGPT in their teaching and assessment practices obtained through a survey in order to establish the degree of their familiarity with the possibilities of ChatGPT for teaching a foreign language and the mechanics of AI-generated test tasks design. The participants are mostly experienced English language instructors, affiliated with higher education institutions in Ukraine, the EU and the USA. Findings reveal varying levels of respondents’ confidence in ChatGPT's functionalities. A considerable proportion of educators appeared to be either unfamiliar with ChatGPT or possess limited knowledge of its capabilities. Hesitancy in implementing ChatGPT in teaching practice primarily stems from a lack of guidance as to its effective utilization closely followed by academic integrity concerns. However, educators who have employed ChatGPT recognize its potential as an assistant in lesson planning, content creation, task design, and assessment, especially for vocabulary, grammar, and writing. Despite acknowledging the tool's benefits, respondents express reservations about ChatGPT-generated tasks’ accuracy for assessment purposes. Challenges include the need for meticulous proofreading, issues with biased information, and insufficient task complexity. Nonetheless, overall satisfaction levels among educators implementing ChatGPT range from moderate to high, substantiating its value in foreign language teaching. Thus the survey stage of the research revealed the growing interest in ChatGPT’s potential for foreign language teaching and assessment among Ukrainian and international EFL and ESP instructors and underscored the need for guidelines and recommendations to create effective ChatGPT-generated test tasks. Future research will develop criteria for evaluating ChatGPT-generated tasks, provide prompt design guidelines, and formulate ethical policies for AI in assessment, addressing educators' concerns and
References
A. Kukulska-Hulme, E. Beirne, G. Conole, E. Costello, T. Coughlan, R. Ferguson, E. FitzGerald, M. Gaved, Ch. Herodotou, W. Holmes, C. Mac Lochlainn, M. N. G. Mhichíl, B. Rienties, J. Sargent, E. Scanlon, M. Sharples, D. Whitelock . “Innovating Pedagogy 2020: Open University Innovation Report”, Institute of Education and Technology (The UK), National Institute for Digital Learning, Dublin City, Ireland, p. 10, 2020. [Online]. Available: https://www.researchgate.net/publication/338991922_Innovating_Pedagogy_2020_Open_University_Innovation_Report_8#fullTextFileContent
S. Sinha, L. Burd, J. Du. Preez, “How ChatGPT could revolutionize academia”, IEEE Spectrum, Feb. 2023. [Online]. Available: https://spectrum.ieee.org/how-chatgpt-could-revolutionize-academia
G. Andersson, N. Titov, B. F. Dear, A. Rozental, P. Carlbring, “Internet-delivered psychological treatments: from innovation to implementation”, World Psychiatry, vol. 18 (1), 2019, pp. 20–28. [Online]. Available: https://onlinelibrary.wiley.com/doi/10.1002/wps.20610
B. Biron, “Online mental health company uses ChatGPT to help respond to users in Experiment – raising ethical concerns around healthcare and AI technology”, Business Insider, 2023. [Online]. Available: https://www.businessinsider.com/company-using-chatgpt-mental-health-support-ethical-issues-2023-1.
A. Sarraju, D. Bruemmer, E. Van Iterson, L. Cho, F. Rodriguez, L. Laffin, “Appropriateness of Cardiovascular Disease Prevention Recommendations Obtained From a Popular Online Chat-Based Artificial Intelligence Model”, JAMA, vol. 329, pp. 842–844, 2023. doi:10.1001/jama.2023.1044.
P. Niszczota, S. Abbas, “GPT has become financially literate: Insights from financial literacy tests of GPT and a preliminary test of how people use it as a source of advice”, Finance Research Letters, vo. 58 (Part A), no. 104333, 2023. [Online]. Available: https://www.sciencedirect.com/science/article/abs/pii/S1544612323007055?via%3Dihub
S. Kreps, M. Jakesch, “Can AI Communication Tools Increase Legislative Responsiveness and Trust in Democratic Institutions?”, Government Information Quarterly, vol. 40(3), no 101829, June 2023, [Online]. Available: https://www.sciencedirect.com/science/article/abs/pii/S0740624X23000291#preview-section-cited-by
S. Steenbergen-Hu, H. Cooper, “A Meta-analysis of the Effectiveness of Intelligent Tutoring Systems on College Students’ Academic Learning”, Journal of Educational Psychology, vol.106, pp. 331–347, 2014.
М. Montebello, AI injected e-learning: The future of online education, Springer, 2018, doi:10.1007/978-3-319-67928-0.
D.-C. Arghir, “Implementation of Learning Management Systems with Generative Artificial Intelligence Functions in the Post-pandemic Environment”, Information Technologies and Learning Tools, vol. 100, no 2, 2024 [Online]. Available: https://journal.iitta.gov.ua/index.php/itlt/issue/view/129
O. Spivakovsky, S. Omelchuk, V. Kobets, Nataliia V. Valko, Daria S. Malchykova, “Institutional Policies on Artificial Intelligence in University Learning, Teaching and Research”, Information Technologies and Learning Tools, vol 97, no 5, 2023. [Online]. Available: https://journal.iitta.gov.ua/index.php/itlt/issue/view/126
O. Synekop, I. Lytovchenko, Y. Lavrysh, V. Lukianenko, “Use of Chat GPT in English for Engineering Classes: Are Students’ and Teachers’ Views on Its Opportunities and Challenges Similar?” International Journal of Interactive Mobile Technologies (iJIM), 18(3), pp. 129–146, (2024). https://doi.org/10.3991/ijim.v18i03.45025.
A. Shiri, “ChatGPT and Academic Integrity”, Information Matters, vol. 3, no 2, Feb. 2023. [Online]. Available: https://ssrn.com/abstract=4360052
K. Huang, “Alarmed by AI Chatbots, Universities Start Revamping How They Teach”, New York Times, Jan. 16, 2023. [Online]. Available: https://www.nytimes.com/2023/01/16/technology/chatgpt-artificial-intelligence-universities.html
W. Liang, M. Yuksekgonul, Y. Mao, E. Wu, J. Zou, “GPT detectors are biased against non-native English writers”, ArXiv, vol. 2, Jul. 2023. [Online]. Available: https://arxiv.org/pdf/2304.02819.pdf.
A. Herft, A Teacher's Prompt Guide to ChatGPT: Aligned with 'What Works Best'. CESE NSW "What Works Best in Practice,” 2023. [Online]. Available: https://drive.google.com/file/d/15qAxnUzOwAPwHzoaKBJd8FAgiOZYcIxq/view?fbclid=IwAR2fRdL5ggq4zU-81FiI8j4BAOp5HqWHC_Ecy2sqKk4EiWXL0FKa5GVz5dE
A. Tlili, B. Shehata, M.A. Adarkwah, A. Bozkurt, D.T. Hickey, R. Huang, B. Agyemang, “What if the devil is my guardian angel: ChatGPT as a case study of using chatbots in education”, Smart Learning Environments, vol. 10(1), no 15, 2023. [Online]. Available:https://link.springer.com/article/10.1186/s40561-023-00237-x?utm_source=getftr&utm_medium=getftr&utm_campaign=getftr_pilot
J.L. Steele, “To GPT or not GPT? Empowering our Students to Learn with AI”, Computers and Education: Artificial Intelligence, vol. 5, no. 100160 2023. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S2666920X23000395
I. Bogost, “ChatGPT is dumber than you think”, The Atlantic, Dec. 2022. [Online]. Available: https://www.theatlantic.com/technology/archive/2022/12/chatgpt-openai-artificial-intelligence-writing-ethics/672386/
C. Newport, “What kind of mind does ChatGPT have?”, The New Yorker, April 2023. [Online]. Available: https://www.newyorker.com/science/annals-of-artificial-intelligence/what-kind-of-mind-does-chatgpt-have
M. Schneider, “Do we need humans anymore?”, U.S. Department of Education, Institute of Education Sciences. Jan. 2023, [Online]. Available: https://ies.ed.gov/director/remarks/01-04-2023.asp
R. Luckin, K. George, and M. Cukurova, “AI for school teachers”, London, UK: CRC Press, 2022.
S. Seufert, J. Guggemos, M. Sailer, “Technology-related knowledge, skills, and attitudes of pre-and in-service teachers: The current situation and emerging trends”, Computers in Human Behavior, vol. 115, no. 106552, Feb. 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0747563220303022
S. Kim, Y. Jang, S. Choi, W. Kim, H. Jung, S. Kim, H. Kim, “Analyzing teacher competency with TPACK for K-12 AI education” KI-Künstliche Intelligenz, vol. 35 (2), pp. 139–151, June 2021. [Online]. Available: https://pure.korea.ac.kr/en/publications/analyzing-teacher-competency-with-tpack-for-k-12-ai-education
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright (c) 2024 Inna Zaiarna, Olena Zhyhadlo, Olha Dunaievska