Keywords
AI chatbots
Research practice
Evaluation of AI tools
AI-powered literary source search
AI analysis of academic sources
How to Cite
Abstract
In modern research practice, AI chatbots are gaining recognition as effective digital assistants that facilitate the search and analysis of scientific literature. In this context, the article presents a comprehensive analysis of the features of using AI chatbots in research practice, particularly for searching and analysing scientific sources. The research methodology is based on a mixed approach that combines a desk review of scientific sources with a comparative assessment of results from 5 AI chatbots (ChatGPT, Microsoft Copilot, Gemini, Claude, Perplexity AI) across 8 criteria. The analysis used quantitative and qualitative metrics to assess AI chatbot responses to the same set of research queries by experts from four fields of knowledge (Engineering, Natural Sciences, Humanities, Social Sciences). A comparative analysis of AI chatbots showed their different capabilities and limitations in working with scientific sources. ChatGPT is the most balanced tool, with high accuracy and objective responses, but it does not cover many sources and is prone to interpretation. MS Copilot is distinguished by the speed of response generation and the sufficient relevance of its sources but is inferior in the objectivity of its analysis. Gemini provides sufficient correspondence with the topic and objectivity in response, but the responses are often excessively long and less accurate in analysis. Claude demonstrates the highest indicators of relevance and objectivity by working with a small number of sources, yet makes almost no generalisations. Perplexity AI is the leader in terms of the volume and quality of the source base, forms generalisations well, but is characterised by the lowest relevance of sources and objectivity of response. The results showed that all the studied AI chatbots demonstrate a high level of formal reliability of sources, but differ significantly in the relevance of sources, analytical depth, and the ability to interpret the scientific content of articles adequately. AI chatbots are effective tools for initial understanding of a research topic, rapid collection of sources for analysis and preliminary structuring of the material. However, they cannot fully carry out a deep and critical analysis of scientific sources on a specific research question.
References
[1] E. Navajas Cawood, M. Vespe, A. Kotsev, and R. Van Bavel, Generative AI Outlook Report: Exploring the Intersection of Technology, Society and Policy. Luxembourg: Publications Office of the European Union, 2025, doi: 10.2760/1109679. (in English)
[2] OECD, OECD Digital Education Outlook 2023: Towards an Effective Digital Education Ecosystem. Paris, France: OECD Publishing, 2023, doi: 10.1787/c74f03de-en. (in English)
[3] UNESCO, Guidance for Generative AI in Education and Research. Paris, France: UNESCO eBooks, 2023, doi: 10.54675/ewzm9535. (in English)
[4] R. Borah, A. W. Brown, P. L. Capers, and K. A. Kaiser, “Analysis of the time and workers needed to conduct systematic reviews of medical interventions using data from the PROSPERO registry,” BMJ Open, vol. 7, no. 2, 2017, Art. no. e012545, doi: 10.1136/bmjopen-2016-012545. (in English)
[5] G. Wagner, R. Lukyanenko, and G. Paré, “Artificial intelligence and the conduct of literature reviews,” J. Inf. Technol., vol. 37, no. 2, pp. 209–226, 2022, doi: 10.1177/02683962211048201. (in English)
[6] E.-M. Schön, J. Kollmorgen, M. Neumann, and M. Rauschenberger, “A case study on using generative AI in literature reviews: Use cases, benefits, and challenges,” in Proc. 21st Int. Conf. Web Information Systems and Technologies (WEBIST), 2025, pp. 533–543, doi: 10.5220/0013672700003985. (in English)
[7] Y. Li et al., “Enhancing systematic literature reviews with generative artificial intelligence: Development, applications, and performance evaluation,” J. Amer. Med. Inform. Assoc., vol. 32, no. 4, pp. 616–625, 2025, doi: 10.1093/jamia/ocaf030. (in English)
[8] F. Bolaños, A. Salatino, F. Osborne, and E. Motta, “Artificial intelligence for literature reviews: Opportunities and challenges,” arXiv preprint arXiv:2402.08565, 2024, doi: 10.48550/arXiv.2402.08565. (in English)
[9] B. Burger, D. K. Kanbach, S. Kraus, M. Breier, and V. Corvello, “On the use of AI-based tools like ChatGPT to support management research,” Eur. J. Innov. Manage., vol. 26, no. 7, pp. 233–241, 2023, doi: 10.1108/EJIM-02-2023-0156. (in English)
[10] M. Rahman, H. J. R. Terano, N. Rahman, A. Salamzadeh, and S. Rahaman, “ChatGPT and academic research: A review and recommendations based on practical examples,” J. Educ., Manage. Develop. Stud., vol. 3, no. 1, pp. 1–12, 2023, doi: 10.52631/jemds.v3i1.175. (in English)
[11] M. Alshater, “Exploring the role of artificial intelligence in enhancing academic performance: A case study of ChatGPT,” SSRN working paper, 2022, doi: 10.2139/ssrn.4312358.. (in English)
[12] Y. N. Gwon et al., “The use of generative AI for scientific literature searches for systematic reviews: ChatGPT and Microsoft Bing AI performance evaluation,” JMIR Med. Inform., vol. 12, 2024, Art. no. e51187, doi: 10.2196/51187. (in English)
[13] M. Sparkman and A. Witt, “Claude AI and Literature Reviews: An Experiment in Utility and Ethical use,” Library Trends, vol. 73, no. 3, pp. 355–380, Feb. 2025, doi: 10.1353/lib.2025.a961199. (in English)
[14] J. Miao, C. Thongprayoon, S. Suppadungsuk, O. A. Garcia Valencia, F. Qureshi, and W. Cheungpasitporn, “Ethical dilemmas in using AI for academic writing and an example framework for peer review in nephrology academia: A narrative review,” Clin. Pract., vol. 14, no. 1, pp. 89–105, 2024, doi: 10.3390/clinpract14010008. (in English)
[15] M. S. Ya’u and M. S. Mohammed, “AI-Assisted Writing and Academic Literacy: Investigating the dual impact of language models on writing proficiency and ethical concerns in Nigerian higher education,” International Journal of Education and Literacy Studies, vol. 13, no. 2, pp. 593–604, Apr. 2025, doi: 10.7575/aiac.ijels.v.13n.2p.593. (in English)
[16] C. Zielinski et al., “Chatbots, generative AI, and scholarly manuscripts: WAME recommendations on chatbots and generative artificial intelligence in relation to scholarly publications,” Curr. Med. Res. Opin., vol. 40, no. 1, pp. 11–13, 2024, doi: 10.1080/03007995.2023.2286102. (in English)
[17] E. S. Rentier, “To use or not to use: Exploring the ethical implications of using generative AI in academic writing,” AI Ethics, vol. 5, pp. 3421–3425, 2025, doi: 10.1007/s43681-024-00649-6. (in English)
[18] E. N. Rabbianty, S. Azizah, and N. K. Virdyna, “AI in academic writing: Assessing current usage and future implications,” INSANIA: J. Pemikiran Alternatif Kependidikan, vol. 28, no. 1a, pp. 14–35, 2023, doi: 10.24090/insania.v28i1a.9278. (in English)
[19] H. Li and X. Wu, “The use of generative AI tools in academic writing: A systematic review of research trends and thematic insights,” AI Ethics, vol. 5, pp. 5821–5840, 2025, doi: 10.1007/s43681-025-00827-0. (in English)
[20] C. D. Manning, P. Raghavan, and H. Schütze, Introduction to Information Retrieval. Cambridge, U.K.: Cambridge Univ. Press, 2008, doi: 10.1017/CBO9780511809071. (in English)
[21] E. Purssell and N. McCrae, How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students. 2024. doi: 10.1007/978-3-031-71159-6. (in English)
[22] L. Elder and R. Paul, Critical thinking: Tools for Taking Charge of Your Learning and Your Life. Bloomsbury Publishing PLC, 2020. (in English)
[23] C. K. Reddy and P. Shojaee, “Towards scientific discovery with generative AI: Progress, opportunities, and challenges,” in Proc. 39th AAAI Conf. Artif. Intell., vol. 39, 2025, pp. 28601–28609, doi: 10.1609/aaai.v39i27.35084. (in English)
[24] L. Kugler, “How do you measure AI?” Commun. ACM, vol. 68, no. 4, pp. 15–17, Apr. 2025, doi: 10.1145/3708972. (in English)
[25] N. Shumeiko, K. Osadcha, and M. Spišiaková, Why artificial intelligence? AI Tools in Information Technology and Foreign Language Education at the Tertiary Level. Berlin, Germany: Peter Lang Verlag, 2026, doi: 10.3726/b23215. (in English)
[26] UNESCO, “Recommendation on the ethics of artificial intelligence,” UNESCO, Paris, France, 2022. [Online]. Available: https://unesdoc.unesco.org/ark:/48223/pf0000381137. (in English)
[27] S. Porsdam Mann et al., “Guidelines for ethical use and acknowledgement of large language models in academic writing,” Nat. Mach. Intell., vol. 6, pp. 1272–1274, 2024, doi: 10.1038/s42256-024-00922-7. (in English)
[28] European Commission, “Ethics guidelines for trustworthy AI,” Digital Strategy, Apr. 08, 2019. [Online]. Available: https://digital-strategy.ec.europa.eu/en/library/ethics-guidelines-trustworthy-ai. (in English)
[29] E. Bailyn, “Top generative AI chatbots by market share – December 2025.” First Page Sage. https://firstpagesage.com/reports/top-generative-ai-chatbots (accessed Feb. 18, 2026). (in English)
[30] P. P. Ray, “ChatGPT: A comprehensive review on background, applications, key challenges, bias, ethics, limitations and future scope,” Internet Things Cyber-Phys. Syst., vol. 3, pp. 121–154, 2023, doi: 10.1016/j.iotcps.2023.04.003. (in English)
[31] T. Wang and Q. Zhu, “ChatGPT – Technical research model, capability analysis, and application prospects,” in Proc. IEEE 7th Adv. Inf. Technol., Electron. Automat. Control Conf. (IAEAC), Chongqing, China, 2024, pp. 787–796, doi: 10.1109/IAEAC59436.2024.10503756. (in English)
[32] S. A. Lehr, A. Caliskan, S. Liyanage, and M. R. Banaji, “ChatGPT as research scientist: Probing GPT’s capabilities as a research librarian, research ethicist, data generator, and data predictor,” Proc. Nat. Acad. Sci., vol. 121, no. 35, 2024, Art. no. e2404328121. (in English)
[33] Microsoft Tech Community, “Reasoning models in Microsoft Copilot: Who’s doing the thinking?” https://techcommunity.microsoft.com/discussions/microsoft365copilot/reasoning-models-in-microsoft-copilot-who%E2%80%99s-doing-the-thinking/4429215 (accessed Feb. 18, 2026). (in English)
[34] Microsoft, “Copilot keeps getting better.” Microsoft.com. https://www.microsoft.com/en-nz/microsoft-copilot/for-individuals?form=MY02P9 (accessed Feb. 18, 2026). (in English)
[35] K. Osadcha, V. Osadchyi, V. Proshkin, and O. Spirin, “Studying IT educators’ satisfaction with using Microsoft Copilot chat to perform professional tasks,” Inf. Technol. Learn. Tools, vol. 107, no. 3, pp. 153–167, 2025, doi: 10.33407/itlt.v107i3.6184. (in English)
[36] N. Shazeer, “Fast transformer decoding: One write-head is all you need,” arXiv preprint arXiv:1911.02150, 2019, doi: 10.48550/arXiv.1911.02150. (in English)
[37] E. Supriyadi, “Exploring Google Bard’s (Gemini) role in enhancing research articles in computational thinking and mathematics education,” Papanda J. Math. Sci. Res., vol. 3, no. 1, pp. 28–37, 2024, doi: 10.56916/pjmsr.v3i1.707. (in English)
[38] Anthropic, “What’s new in Claude 4.5.” Anthropic.com. https://platform.claude.com/docs/en/about-claude/models/whats-new-claude-4-5 (accessed Jan. 01, 2026). (in English)
[39] Anthropic, “System card: Claude Opus 4.5,” 2025. Anthropic.com. https://assets.anthropic.com/m/64823ba7485345a7/Claude-Opus-4-5-System-Card.pdf (accessed Jan. 01, 2026). (in English)
[40] Perplexity, “Getting started. What is Perplexity?” Perplexity.ai. https://www.perplexity.ai/hub/getting-started (accessed Jan. 01, 2026). (in English)
[41] Perplexity, “What advanced AI models are included in my subscription?” Perplexity.ai. https://www.perplexity.ai/help-center/en/articles/10354919-what-advanced-ai-models-are-included-in-my-subscription (accessed Jan. 01, 2026). (in English)
[42] Perplexity, “Academic filter guide.” Perplexity.ai. https://docs.perplexity.ai/guides/academic-filter-guide (accessed Jan. 01, 2026). (in English)
[43] J. Khanifar, “Evaluating AI-generated responses from different chatbots to soil science-related questions,” Soil Adv., vol. 3, 2025, Art. no. 100034, doi: 10.1016/j.soilad.2025.100034. (in English)
[44] M. F. Şahin et al., “Responses of five different artificial intelligence chatbots to the top searched queries about erectile dysfunction: A comparative analysis,” J. Med. Syst., vol. 48, 2024, Art. no. 38, doi: 10.1007/s10916-024-02056-0. (in English)
[45] E. Bostan, M. T. Uçar, and E. Dönmez, “Evaluation of the accuracy, reliability, quality, and readability of artificial intelligence chatbots-generated responses to acne-related questions,” Turk. J. Dermatol., vol. 19, no. 4, pp. 235–243, 2025, doi: 10.4274/tjd.galenos.2025.06977. (in English)
[46] Perplexity, “What is research mode?” Perplexity.ai. https://www.perplexity.ai/help-center/en/articles/10738684-what-is-research-mode (accessed Jan. 01, 2026). (in English)
[47] K. Osadcha and V. Osadchyi, “Dataset of AI Chatbot Responses to Four Research Questions”. Zenodo, May 05, 2026. doi: 10.5281/zenodo.20038076. (in English)
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright (c) 2026 Kateryna Osadcha, Maryna Osadcha, Natalia Shumeiko, Volodymyr Proshkin