Keywords
effectiveness
criteria and indicators
publication
dissemination
impact
expert evaluation
How to Cite
Abstract
The article studies the problem of evaluating the results of scientific and pedagogical research. The study outlines the various advantages of using the altmetric approach, such as obtaining various indicators to assess the impact of research, overcoming disciplinary differences, and attracting a wider readership. The article analyzes altmetric indicators of the effectiveness of scientific research, in particular, based on views, citations, bookmarking and discussion. Analyzing the existing models for assessing research effectiveness, the authors conclude that they need to be improved and adapted to the case of pedagogical research conducted by teams of scientists. Accordingly, the authors’ model of using information and digital technologies to evaluate the effectiveness of collective pedagogical research includes target, content, organizational, activity and evaluation and result blocks. It is indicated that the evaluation of the effectiveness of collective pedagogical research is an additive value formed as the sum of the evaluation of publication, dissemination, impact and expert evaluation. The indicators of the publication criterion are the number of publications in international scientometric and open bibliometric databases, in scientific electronic libraries, academic and public social networks. Dissemination indicators include the number of downloads, archiving, and follow-up of research conducted within the collective theme. Impact indicators include citations of publications in the above publications, dissemination of research results, comments, reactions, and citations in public social networks, views, comments, and citations in academic social networks. The indicators of the expert criterion include the novelty and validity of the results obtained, the appropriateness and validity of the research methods used, its long-term impact on the field of education, the possibility of implementing the results in practice, attracting additional sources of funding, and the quality of publications of the research leader and research performers. The article provides examples of the use of scientometric services and digital tools to assess the effectiveness of the given criteria.
References
Верховна Рада України. (26.11.2015). Закон України № 848-VIII, Про наукову і науково-технічну діяльність. Дата звернення: 10 жовт. 2023. [Електронний ресурс]. Доступно: https://zakon.rada.gov.ua/laws/show/848-19#Text
В. Ю. Биков, О. М. Спірін, С. М. Іванова, Т. А. Вакалюк, І. С. Мінтій та А. В. Кільченко, “Наукометричні показники оцінювання результативності досліджень наукових установ і закладів освіти”, ІТЗН, т. 86, № 6, с. 289–312, груд. 2021. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.33407/itlt.v86i6.4656.
В. Ю. Биков, О. М. Спірін, А. О. Білощицький, О. Ю. Кучанський, О. В. Діхтяренко та О. В. Новицький, “Відкриті цифрові системи в оцінюванні результатів науково-педагогічних досліджень”, ІТЗН, т. 75, № 1, с. 294–315, лют. 2020. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.33407/itlt.v75i1.3589.
S. Kvitka, H. Starushenko та I. Andrianov, “Scientometric methods of analysis of the effectiveness of scientific research according to Web of Sciense”, Public admin. aspects, т. 8, № 1 SI, с. 60–67, лип. 2020. Дата звернення: 10 жовт. 2023 doi: https://doi.org/10.15421/152042.
M. Gusenbauer та N. R. Haddaway, “Which academic search systems are suitable for systematic reviews or meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources”, Res. Synthesis Methods, т. 11, № 2, с. 181–217, січ. 2020. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1002/jrsm.1378.
V. GrigasSimona, S. Juzėnienė та J. Veličkaitė, “‘Just Google it’ – the scope of freely available information sources for doctoral thesis writing”, Inf. Res. int. electron. j., т. 22, № 1, 2017.
G. Halevi, H. Moed та J. Bar-Ilan, “Suitability of Google Scholar as a source of scientific information and as a source of data for scientific evaluation—Review of the Literature”, J. Informetrics, т. 11, № 3, с. 823–834, серп. 2017. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1016/j.joi.2017.06.005.
J. Mingers та M. Meyer, “Normalizing Google Scholar data for use in research evaluation”, Scientometrics, т. 112, № 2, с. 1111–1121, трав. 2017. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1007/s11192-017-2415-x.
В. П. Олексюк “Досвід інтеграції хмарних сервісів google apps у інформаційно-освітній простір вищого навчального закладу”, ІТЗН, т. 35, № 3, с. 64–73, черв. 2013. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.33407/itlt.v35i3.824.
J. A. Moral-Muñoz, E. Herrera-Viedma, A. Santisteban-Espejo та M. J. Cobo, “Software tools for conducting bibliometric analysis in science: An up-to-date review”, El Prof. Informacion, т. 29, № 1, січ. 2020. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.3145/epi.2020.ene.03.
X. Pan, E. Yan, M. Cui та W. Hua, “Examining the usage, citation, and diffusion patterns of bibliometric mapping software: A comparative study of three tools”, J. Informetrics, т. 12, № 2, с. 481–493, трав. 2018. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1016/j.joi.2018.03.005.
L. Bornmann, “Scientific peer review”, Annu. Rev. Inf. Sci. Technol., т. 45, № 1, с. 197–245, 2011. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1002/aris.2011.1440450112.
M. Teplitskiy, D. Acuna, A. Elamrani-Raoult, K. Körding та J. Evans, “The sociology of scientific validity: How professional networks shape judgement in peer review”, Res. Policy, т. 47, № 9, с. 1825–1841, листоп. 2018. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1016/j.respol.2018.06.014.
Z. Krykhovetska, S. Kropelnytska та O. Kondur, “Project activities as a factor of improving competitiveness of domestic higher educational establishments (HEEs)”, Pryazovskyi Econ. Herald, № 6(17), 2019. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.32840/2522-4263/2019-6-28.
S. O. Semerikov, O. M. Spirin, T. A. Vakaliuk, I. S. Mintii, S. M. Ivanova та O. M. Shymon, “Evaluating the effectiveness of pedagogical research: identifying criteria and indicators for academic and research staff”, Educ. Dimens., черв. 2023. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.31812/ed.605.
J. Priem, P. Groth та D. Taraborelli, “The Altmetrics Collection”, PLoS ONE, т. 7, № 11, листоп. 2012, ст. № e48753. Дата звернення: 18 жовт. 2023. doi: https://doi.org/10.1371/journal.pone.0048753.
M. Taylor, “An altmetric attention advantage for open access books in the humanities and social sciences”, Scientometrics, т. 125, № 3, с. 2523–2543, жовт. 2020. Дата звернення: 18 жовт. 2023. doi: https://doi.org/10.1007/s11192-020-03735-8.
H. Yu, B. Murat, L. Li та T. Xiao, “How accurate are Twitter and Facebook altmetrics data? A comparative content analysis”, Scientometrics, т. 126, № 5, с. 4437–4463, берез. 2021. Дата звернення: 18 жовт. 2023. doi: https://doi.org/10.1007/s11192-021-03954-7.
M. Shahzad, C. Freeman, M. Rahimi та H. Alhoori, “Predicting Facebook sentiments toward research”, Natural Lang. Process. J., с. 100010, трав. 2023. Дата звернення: 17 жовт. 2023. doi: https://doi.org/10.1016/j.nlp.2023.100010.
L. Bornmann, “Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics”, J. Informetrics, т. 8, № 4, с. 895–903, жовт. 2014. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1016/j.joi.2014.09.005.
M. Thelwall, S. Haustein, V. Larivière та C. R. Sugimoto, “Do Altmetrics Work? Twitter and Ten Other Social Web Services”, PLoS ONE, т. 8, № 5, трав. 2013, ст. № e64841. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1371/journal.pone.0064841.
H. Shema, J. Bar-Ilan та M. Thelwall, “Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics”, J. Assoc. Inf. Sci. Technol., т. 65, № 5, с. 1018–1027, січ. 2014. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1002/asi.23037.
S. Haustein, R. Costas та V. Larivière, “Characterizing Social Media Metrics of Scholarly Papers: The Effect of Document Properties and Collaboration Patterns”, PLOS ONE, т. 10, № 3, берез. 2015, ст. № e0120495. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1371/journal.pone.0120495.
O. B. Onyancha, “Research Excellence in the Era of Online Attention: Altmetrics of South Africa’s Highly Cited Papers in Selected Research Fields”, Publishing Res. Quart., т. 36, № 1, с. 169–185, серп. 2019. Дата звернення: 18 жовт. 2023. doi: https://doi.org/10.1007/s12109-019-09679-z.
X. Huang, X. Liu, Y. Shang, F. Qiao та G. Chen, “Current Trends in Research on Bone Regeneration: A Bibliometric Analysis”, BioMed Res. Int., т. 2020, с. 1–12, трав. 2020. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1155/2020/8787394.
E. Ferrara та A. E. Romero, “Scientific impact evaluation and the effect of self-citations: Mitigating the bias by discounting the h-index”, J. Amer. Soc. Inf. Sci. Technol., т. 64, № 11, с. 2332–2339, черв. 2013. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1002/asi.22976.
J. L. Ortega, “Blogs and news sources coverage in altmetrics data providers: a comparative analysis by country, language, and subject”, Scientometrics, т. 122, № 1, с. 555–572, листоп. 2019. Дата звернення: 18 жовт. 2023. doi: https://doi.org/10.1007/s11192-019-03299-2.
E. Mohammadi та M. Thelwall, “Assessing non-standard article impact using F1000 labels”, Scientometrics, т. 97, № 2, с. 383–395, берез. 2013. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1007/s11192-013-0993-9.
R. Haunschild та L. Bornmann, “Normalization of Mendeley reader counts for impact assessment”, J. Informetrics, т. 10, № 1, с. 62–73, лют. 2016. Дата звернення: 17 жовт. 2023. doi: https://doi.org/10.1016/j.joi.2015.11.003.
L. Waltman та R. Costas, “F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations”, J. Assoc. Inf. Sci. Technol., т. 65, № 3, с. 433–445, жовт. 2013. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1002/asi.23040.
R. Yoshimura, M. C. Grant, M. D. Gardiner та R. G. Wade, “Disseminating Hand Surgery Research Using Social Media: The Relationship Between Altmetrics and Citations”, J. Hand Surgery, т. 46, № 9, с. 740–747, верес. 2021. Дата звернення: 10 жовт. 2023. doi: https://doi.org/10.1016/j.jhsa.2021.03.028.
О. М. Спірін, Т. А. Вакалюк, В. П. Олексюк, С. М. Іванова, . І. С. Мінтій, і А. В. Кільченко, «Модель використання інформаційно-цифрових технологій для оцінювання результативності педагогічних досліджень», OpenEdu, вип. 14, с. 50–62, Квіт 2023.
С. Гончаренко, “Про критерії оцінювання педагогічних досліджень”, Педагогіка і психологія проф. освіти, т. 5, с. 80–89, 1998. Дата звернення: 18 жовт. 2023. [Електронний ресурс]. Доступно: https://lib.iitta.gov.ua/706520/.
F. Franceschini та D. Maisano, “Aggregating multiple ordinal rankings in engineering design: the best model according to the Kendall’s coefficient of concordance”, Res. Eng. Des., жовт. 2020. Дата звернення: 18 жовт. 2023. doi:https://doi.org/10.1007/s00163-020-00348-3.
REFERENCES (TRANSLATED AND TRANSLITERATED)
Verkhovna Rada of Ukraine. (26.11.2015). Law of Ukraine No. 848-VIII, On Scientific and Scientific-Technical Activities. [Online]. Available: https://zakon.rada.gov.ua/laws/show/848-19#Text (in Ukrainian)
V. Yu. Bykov, O. M. Spirin, S. I. Ivanova, T. A. Vakaliuk, I. S. Mintii and A. V. Kilchenko, “Scientometric Indicators for Evaluating the Effectiveness of Pedagogical Research of Scientific Institutions and Educational Institutions”, Information Technologies and Learning Tools, vol. 86, no. 6, pp. 289–312, doi: https://doi.org/10.33407/itlt.v86i6.4656. Accessed on: Feb. 10, 2023 (in Ukrainian)
V. Yu. Bykov, O. M. Spirin, A. O. Biloshchytskyi, O. Yu. Kuchansky, O. V. Dikhtiarenko and O. V. Novytskyi, “Open Digital Systems for Assessment of Pedagogical Research Results”, Information Technologies and Learning Tools vol. 75, no. 1, pp. 294–315, doi: https://doi.org/10.33407/itlt.v75i1.3589. Accessed on: Feb. 10, 2023. (in Ukrainian)
S. Kvitka, H. Starushenko and I. Andrianov, “Scientometric methods of analysis of the effectiveness of scientific research according to Web of Sciense”, Public admin. aspects, vol. 8, no 1 SI, pp. 60–67, doi: https://doi.org/10.15421/152042. (in Ukrainian)
M. Gusenbauer and N. R. Haddaway, “Which Academic Search Systems are Suitable for Systematic Reviews or Meta‐analyses? Evaluating retrieval qualities of Google Scholar, PubMed, and 26 other resources”, Res. Synthesis Methods, vol. 11, no 2, pp. 181–217, doi: https://doi.org/10.1002/jrsm.1378. (in English)
V. GrigasSimona, S. Juzėnienė and J. Veličkaitė, “‘Just Google it’ – the Scope of Freely Available Information Sources for Doctoral Thesis Writing”, Inf. Res. int. electron. j., vol. 22, no 1, 2017. (in English)
G. Halevi, H. Moed and J. Bar-Ilan, “Suitability of Google Scholar as a Source of Scientific Information and as a Source of Data for Scientific Evaluation—Review of the Literature”, J. Informetrics, vol. 11, no 3, pp. 823–834, doi: https://doi.org/10.1016/j.joi.2017.06.005. (in English)
J. Mingers and M. Meyer, “Normalizing Google Scholar Data for Use in Research Evaluation”, Scientometrics, T. 112, no 2, pp. 1111–1121, doi: https://doi.org/10.1007/s11192-017-2415-x. (in English)
V. P. Oleksyuk, “Experience of The Integration of Cloud Services Google Apps into Information And Educational Space of Higher Educational Institution”, Information Technologies and Learning Tools, vol. 35, no. 3, pp. 64–73, doi: https://doi.org/10.33407/itlt.v35i3.824. (in Ukrainian)
J. A. Moral-Muñoz, E. Herrera-Viedma, A. Santisteban-Espejo and M. J. Cobo, “Software tools for Conducting Bibliometric Analysis in Science: An up-to-date Review”, El Prof. Informacion, T. 29, no 1, doi: https://doi.org/10.3145/epi.2020.ene.03. (in English)
X. Pan, E. Yan, M. Cui and W. Hua, “Examining the Usage, Citation, and Diffusion Patterns of Bibliometric Mapping Software: A Comparative Study of Three Tools”, J. Informetrics, vol. 12, no 2, pp. 481–493, doi: https://doi.org/10.1016/j.joi.2018.03.005. (in English)
L. Bornmann, “Scientific Peer Review”, Annu. Rev. Inf. Sci. Technol., vol. 45, no 1, pp. 197–245, 2011. doi: https://doi.org/10.1002/aris.2011.1440450112. (in English)
M. Teplitskiy, D. Acuna, A. Elamrani-Raoult, K. Körding and J. Evans, “The sociology of scientific validity: How professional networks shape judgement in peer review”, Res. Policy, vol. 47, no 9, pp. 1825–1841, doi: https://doi.org/10.1016/j.respol.2018.06.014. (in English)
Z. Krykhovetska, S. Kropelnytska and O. Kondur, “Project activities as a factor of improving competitiveness of domestic higher educational establishments (HEEs)”, Pryazovskyi Econ. Herald, no 6(17), 2019. doi: https://doi.org/10.32840/2522-4263/2019-6-28. (in English)
S. O. Semerikov, O. M. Spirin, T. A. Vakaliuk, I. S. Mintii, S. M. Ivanova and O. M. Shymon, “Evaluating the effectiveness of pedagogical research: identifying criteria and indicators for academic and research staff”, Educ. Dimens., doi: https://doi.org/10.31812/ed.605. (in Ukrainian)
J. Priem, P. Groth та D. Taraborelli, “The Altmetrics Collection”, PLoS ONE, т. 7, № 11, Nov. 2012, p. e48753. doi: https://doi.org/10.1371/journal.pone.0048753. (in English)
M. Taylor, “An altmetric attention advantage for open access books in the humanities and social sciences”, Scientometrics, vol. 125, no 3, pp. 2523–2543, doi: https://doi.org/10.1007/s11192-020-03735-8. (in English)
H. Yu, B. Murat, L. Li та T. Xiao, “How accurate are Twitter and Facebook altmetrics data? A comparative content analysis”, Scientometrics, vol. 126, no 5, pp. 4437–4463, берез. 2021. doi: https://doi.org/10.1007/s11192-021-03954-7. (in English)
M. Shahzad, C. Freeman, M. Rahimi and H. Alhoori, “Predicting Facebook sentiments toward research”, Natural Lang. Process. J., doi: https://doi.org/10.1016/j.nlp.2023.100010. (in English)
L. Bornmann, “Do altmetrics point to the broader impact of research? An overview of benefits and disadvantages of altmetrics”, J. Informetrics, vol. 8, no 4, pp. 895–903, doi: https://doi.org/10.1016/j.joi.2014.09.005. (in English)
M. Thelwall, S. Haustein, V. Larivière and C. R. Sugimoto, “Do Altmetrics Work? Twitter and Ten Other Social Web Services”, PLoS ONE, T. 8, no 5, doi: https://doi.org/10.1371/journal.pone.0064841. (in English)
H. Shema, J. Bar-Ilan and M. Thelwall, “Do blog citations correlate with a higher number of future citations? Research blogs as a potential source for alternative metrics”, J. Assoc. Inf. Sci. Technol., т. 65, no 5, pp. 1018–1027, Jan. 2014. doi: https://doi.org/10.1002/asi.23037. (in English)
S. Haustein, R. Costas and V. Larivière, “Characterizing Social Media Metrics of Scholarly Papers: The Effect of Document Properties and Collaboration Patterns”, PLOS ONE, т. 10, no 3, Mar. 2015. doi: https://doi.org/10.1371/journal.pone.0120495. (in English)
O. B. Onyancha, “Research Excellence in the Era of Online Attention: Altmetrics of South Africa’s Highly Cited Papers in Selected Research Fields”, Publishing Res. Quart., vol. 36, no 1, pp. 169–185, Aug. 2019. doi: https://doi.org/10.1007/s12109-019-09679-z. (in English)
X. Huang, X. Liu, Y. Shang, F. Qiao and G. Chen, “Current Trends in Research on Bone Regeneration: A Bibliometric Analysis”, BioMed Res. Int., T. 2020, pp. 1–12, May. 2020. doi: https://doi.org/10.1155/2020/8787394. (in English)
E. Ferrara and A. E. Romero, “Scientific impact evaluation and the effect of self-citations: Mitigating the bias by discounting the h-index”, J. Amer. Soc. Inf. Sci. Technol., vol. 64, no 11, pp. 2332–2339, Jun. 2013. doi: https://doi.org/10.1002/asi.22976. (in English)
J. L. Ortega, “Blogs and news sources coverage in altmetrics data providers: a comparative analysis by country, language, and subject”, Scientometrics, vol. 122, no 1, pp. 555–572, Nov. 2019. doi: https://doi.org/10.1007/s11192-019-03299-2. (in English)
E. Mohammadi and M. Thelwall, “Assessing non-standard article impact using F1000 labels”, Scientometrics, vol. 97, no 2, pp. 383–395, Mar. 2013. doi: https://doi.org/10.1007/s11192-013-0993-9. (in English)
R. Haunschild and L. Bornmann, “Normalization of Mendeley reader counts for impact assessment”, J. Informetrics, vol. 10, no 1, pp. 62–73, Feb. 2016. doi: https://doi.org/10.1016/j.joi.2015.11.003. (in English)
L. Waltman and R. Costas, “F1000 Recommendations as a Potential New Data Source for Research Evaluation: A Comparison With Citations”, J. Assoc. Inf. Sci. Technol., vol. 65, no 3, pp. 433–445, Oct. 2013. doi: https://doi.org/10.1002/asi.23040. (in English)
R. Yoshimura, M. C. Grant, M. D. Gardiner and R. G. Wade, “Disseminating Hand Surgery Research Using Social Media: The Relationship Between Altmetrics and Citations”, J. Hand Surgery, vol. 46, no. 9, pp. 740–747, Sep. 2021. doi: https://doi.org/10.1016/j.jhsa.2021.03.028. (in English)
O. Spirin, T. Vakaliuk, V. Oleksiuk, S. Ivanova, I. Mintii та A. Kilchenko, “A Model of Using Information and Digital Technologies to Evaluate the Effectiveness of Pedagogical Research”, Open educational e-environ. modern univ., no 14, pp. 50–62, 2023. doi: https://doi.org/10.28925/2414-0325.2023.145. (in Ukrainian)
S. Goncharenko, "On criteria for evaluating pedagogical research", Pedagogy and psychology, prof. of education, vol. 5, pp. 80–89, 1998. [Online]. Available: https://lib.iitta.gov.ua/706520/ (in Ukrainian)
F. Franceschini, D. Maisano. “Aggregating multiple ordinal rankings in engineering design: the best model according to the Kendall’s coefficient of concordance@. Research in Engineering Design. Oct. 2020. doi: https://doi.org/10.1007/s00163-020-00348-3. (in English)
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Copyright (c) 2023 Олег Михайлович Спірін; Василь Петрович Олексюк, Світлана Миколаївна Іванова, Тетяна Леонідівна Новицька