Keywords
uniform DIF
non-uniform DIF
logistic regression
Mantel-Haenszel method
Likelihood Ratio Test
How to Cite
Abstract
The necessary condition for the presence of biased assessment by some test is differential item functioning in different groups of test takers. The ideas of some statistical methods for detecting Differential Item Functioning are described in the given article. They were developed in the framework of the main approaches to modeling test results: using contingency tables, regression models, multidimensional models and models of Item Response Theory. The Mantel-Haenszel procedure, logistic regression method, SIBTEST and Item Response Theory Likelihood Ratio Test are considered. The characteristics of each method and conditions of their application are specified. Overview of existing free software tools implementing these methods is carried out. Comparisons of these methods are conducted on the example of real data. Also notes that it is appropriate to use several methods simultaneously to reduce the risk of false conclusions.References
Лісова Т. В. DIF та DPF аналіз тесту на креативну самодостатність за допомогою програми Winsteps / Тетяна Лісова // Гуманітарний вісник. Тематичний випуск «Міжнародні Челпанівські психолого-педагогічні читання». – 2013. – Додаток 1 до Вип. 29, Том 1. – С. 255–261.
Фройнд Ф. А. Анализ DIF в оценке общего интеллекта для генерируемых компьютером графических тестовых задач в двух этнически различных выборках / [Ф. А. Фройнд, С. В. Давыдов, Й. П. Бертлинг, Х. Холлинг, Г. С. Шляхтин] // Социология. Психология. Философия. Вестник Нижегородского университета им. Н.И. Лобачевского. – 2012. – № 5 (1). – С. 334–341.
Acar T. Comparison of Differential Item Functioning Determination Techniques: HGLM, LR and IRT-LR / T. Acar, H. Kelecioglu // Educational Sciences: Theory & Practice. – 2010. – 10 (2). – P. 639–649.
Ayala R. J. The Theory and Practice of Item Response Theory / R. J. de Ayala. – New York: Guilford Publications Incorporated, 2009. – 448 p.
Bolt D. M. A monte carlo comparison of parametric and nonparametric polytomous DIF detection methods / D. M. Bolt // Applied Measurement in Education. – 2002. – Vol. 15. – P. 113–141.
Camilli G. Test fairness / Gregory Camilli / In R. Brennan (Ed.), Educational measurement. – Westport, CT: ACE, Praeger series on higher education, 2006. – P. 221–256.
Dorans N.J. ETS Contributions to the Quantitative Assessment of Item, Test, and Score Fairness / N.J. Dorans. – Educational Testing Service: Princeton, New Jersey, 2013. – 38 p.
Ercikan K. Comparability of bilingual versions of assessments: Sources of incomparability of English and French versions of Canada’s national achievement tests / [K. Ercikan, M. J. Gierl, T. McCreith et al.[ // Applied Measurement in Education. – 2004. – Vol. 17(3). – P. 301–321.
Ercikan K. Examining the Construct Comparability of the English and French Versions of TIMSS / K. Ercikan, K. Koh // International Journal of Testing. – 2005. – Vol. 5(1). – P. 23–35.
Gierl M.J. Performance of SIBTEST When the Percentage of DIF Items is Large / M. J. Gierl, A. Gotzmann, K. A. Boughton // Applied Measurement in Education. – 2004. – Vol. 17(3). – P. 241–264.
González A. EASY-DIF: Software for Analyzing Differential Item Functioning Using the Mantel-Haenszel and Standardization Procedures / [A. González, J. L. Padilla, M. D. Hidalgo et al.[ // Applied Psychological Measurement. – 2011. – Vol. 35(6). – P. 483–484.
Hambleton R. K. Translating achievement tests for use in cross-cultural studies / R.K. Hambleton // European Journal of Psychological Assessment. – 1993. – Vol. 9. – P. 57–68.
Holland P. W. Differential item performance and the Mantel-Haenszel procedure / P. W. Holland, D. T. In H. Wainer and H. Braun (ed), Test validity. – Hillsdale, NJ: Erlbaum, 1988. – P. 129–145.
Le L.T . Investigating Gender Differential Item Functioning Across Countries and Test Languages for PISA Science Items / Luc T. Le // International Journal of Testing. – 2009. – Vol. 9(2). – P. 122–133.
Oliveri M. E. Effects of Population Heterogeneity on Accuracy of DIF Detection / M. E. Oliveri, K. Ercikan, B. D. Zumbo // Applied Measurement in Education. – 2014. – Vol. 27(4). – P. 286–300.
Sandilands D. Investigating Sources of Differential Item Functioning in International Large-Scale Assessments Using a Confirmatory Approach / [D. Sandilands, M. E. Oliveri, B. D. Zumbo et al.] // International Journal of Testing. – 2013. – Vol. 13. – P. 152–174.
Shealy R. A model-based standardization approach that separates true bias/DIF from group differences and detects test bias/DIF as well as item bias/DIF / R. Shealy, W.F. Stout // Psychometrika. – 1993. – Vol.58. – P. 159–194.
SIBTEST manual [Електронний ресурс] / W. F. Stout, L. Roussos, 1995. – Режим доступу : http://psychometrictools.measuredprogress.org/dif1.
Stoneberg B. D. A study of gender-based and ethnic-based Differential Item Functioning (DIF) in the Spring 2003 Idaho Standards Achievement tests applying the Simultaneous Bias Test (SIBTEST) and the Mantel-Haenszel chi-square Test [Електронний ресурс]. – Режим доступу : http://files.eric.ed.gov/fulltext/ED489949.pdf.
Swaminathan H. Detecting differential item functioning using logistic regression procedures / H. Swaminathan, H. J. Rogers // Journal of Educational Measurement. – 1990. – Vol. 27. – P. 361–370.
Thissen D. Detection of differential item functioning using the parameters of item response models / D. Thissen, L. Steinberg, H. Wainer // In P.W. Holland and H. Wainer (ed), Differential item functioning. – Hillsdale, NJ: Erlbaum, 1993. – P. 67–113.
Thissen, D. IRTLRDIF v.2.0b: Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning [Електронний ресурс]. – Режим доступу : http://www.unc.edu/~dthissen.
Zumbo B. D. A Handbook on the Theory and Methods of Differential Item Functioning (DIF): Logistic Regression Modeling as a Unitary Framework for Binary and Likert-Type (Ordinal) Item Scores [Електронний ресурс]. – Режим доступу : http://www.educ.ubc.ca/faculty/zumbo/DIF/index.html.
Zumbo B. D. Three Generations of DIF Analyses: Considering Where It Has Been, Where It Is Now, and Where It Is Going / B. D. Zumbo // Language Assessment Quarterly. – 2007. – Vol. 4(2). – P. 223–233.
REFERENCES (TRANSLATED AND TRANSLITERATED)
Lisova T. V. DIF and DPF analysis of the test on creative self-sufficiency using Winsteps / Tetiana Lisova // Ghumanitarnyj visnyk. Special issue «International Chelpanov psychological and pedagogical reading». – 2013. – Annex 1 to the Issue 29, Vol. 1. – P. 255–261 (in Ukrainian).
Freund F. A. DIF analysis in the assessment of general intelligence for computer-generated graphics test tasks in two ethnically different samples / F. A. Freund, S. V. Davydov, J. P. Bertling, H. Holling, G. S. Shlyakhtin // Sociologija. Psihologija. Filosofija. Vestnik Nizhegorodskogo universiteta im. N.I. Lobachevskogo. – 2012. – Vol. 5 (1). – P. 334–341 (in Russian).
Acar T. Comparison of Differential Item Functioning Determination Techniques: HGLM, LR and IRT-LR / T. Acar, H. Kelecioglu // Educational Sciences: Theory & Practice. – 2010. – 10 (2). – P. 639–649 (in English).
Ayala R. J. The Theory and Practice of Item Response Theory / R. J. de Ayala. – New York: Guilford Publications Incorporated, 2009. – 448 p. (in English).
Bolt D. M. A monte carlo comparison of parametric and nonparametric polytomous DIF detection methods / D.M. Bolt // Applied Measurement in Education. – 2002. – Vol. 15. – P. 113–141 (in English).
Camilli G. Test fairness / Gregory Camilli / In R. Brennan (Ed.), Educational measurement. – Westport, CT: ACE, Praeger series on higher education, 2006. – P. 221–256 (in English).
Dorans N. J. ETS Contributions to the Quantitative Assessment of Item, Test, and Score Fairness / N. J. Dorans. – Educational Testing Service: Princeton, New Jersey, 2013. – 38 p. (in English).
Ercikan K. Comparability of bilingual versions of assessments: Sources of incomparability of English and French versions of Canada’s national achievement tests / K. Ercikan, M. J. Gierl, T. McCreith et al. // Applied Measurement in Education. – 2004. – Vol. 17(3). – P. 301–321 (in English)..
Ercikan K. Examining the Construct Comparability of the English and French Versions of TIMSS / K. Ercikan, K. Koh // International Journal of Testing. – 2005. – Vol. 5(1). – P. 23–35 (in English).
Gierl M. J. Performance of SIBTEST When the Percentage of DIF Items is Large / M. J. Gierl, A. Gotzmann, K. A. Boughton // Applied Measurement in Education. – 2004. – Vol. 17(3). – P. 241–264 (in English).
González A. EASY-DIF: Software for Analyzing Differential Item Functioning Using the Mantel-Haenszel and Standardization Procedures / A. González, J. L. Padilla, M. D. Hidalgo et al. // Applied Psychological Measurement. – 2011. – Vol. 35(6). – P. 483–484 (in English).
Hambleton R. K. Translating achievement tests for use in cross-cultural studies / R. K. Hambleton // European Journal of Psychological Assessment. – 1993. – Vol. 9. – P. 57–68 (in English).
Holland P. W. Differential item performance and the Mantel-Haenszel procedure / P. W. Holland, D. T. Thayer / In H. Wainer and H. Braun (ed), Test validity. – Hillsdale, NJ: Erlbaum, 1988. – P. 129–145 (in English).
Le L.T. Investigating Gender Differential Item Functioning Across Countries and Test Languages for PISA Science Items / Luc T. Le // International Journal of Testing. – 2009. – Vol. 9(2). – P. 122–133. (in English)
Oliveri M. E. Effects of Population Heterogeneity on Accuracy of DIF Detection / M. E. Oliveri, K. Ercikan, B. D. Zumbo // Applied Measurement in Education. – 2014. – Vol. 27(4). – P. 286–300 (in English).
Sandilands D. Investigating Sources of Differential Item Functioning in International Large-Scale Assessments Using a Confirmatory Approach / D. Sandilands, M. E. Oliveri, B. D. Zumbo et al. // International Journal of Testing. – 2013. – Vol. 13. – P. 152–174 (in English).
Shealy R. A model-based standardization approach that separates true bias/DIF from group differences and detects test bias/DIF as well as item bias/DIF / R. Shealy, W. F. Stout // Psychometrika. – 1993. – Vol.58. – P. 159–194 (in English).
SIBTEST manual [online] / W.F. Stout, L. Roussos, 1995. – Available from : http://psychometrictools.measuredprogress.org/dif1 (in English).
Stoneberg B. D. A study of gender-based and ethnic-based Differential Item Functioning (DIF) in the Spring 2003 Idaho Standards Achievement tests applying the Simultaneous Bias Test (SIBTEST) and the Mantel-Haenszel chi-square Test [online]. – Available from : http://files.eric.ed.gov/fulltext/ED489949.pdf (in English).
Swaminathan H. Detecting differential item functioning using logistic regression procedures / H. Swaminathan, H. J. Rogers // Journal of Educational Measurement. – 1990. – Vol. 27. – P. 361–370 (in English).
Thissen D. Detection of differential item functioning using the parameters of item response models / D. Thissen, L. Steinberg, H. Wainer // In P. W. Holland and H. Wainer (ed), Differential item functioning. – Hillsdale, NJ: Erlbaum, 1993. – P. 67–113 (in English).
Thissen, D. IRTLRDIF v.2.0b: Software for the computation of the statistics involved in item response theory likelihood-ratio tests for differential item functioning [online]. – Available from : http://www.unc.edu/~dthissen (in English)
Zumbo B. D. A Handbook on the Theory and Methods of Differential Item Functioning (DIF): Logistic Regression Modeling as a Unitary Framework for Binary and Likert-Type (Ordinal) Item Scores [online]. – Available from : http://www.educ.ubc.ca/faculty/zumbo/DIF/index.html (in English).
Zumbo B. D. Three Generations of DIF Analyses: Considering Where It Has Been, Where It Is Now, and Where It Is Going / B. D. Zumbo // Language Assessment Quarterly. – 2007. – Vol. 4(2). – P. 223–233 (in English).
Authors who publish in this journal agree to the following terms:
- Authors hold copyright immediately after publication of their works and retain publishing rights without any restrictions.
- The copyright commencement date complies the publication date of the issue, where the article is included in.
Content Licensing
- Authors grant the journal a right of the first publication of the work under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License (CC BY-NC-SA 4.0) that allows others freely to read, download, copy and print submissions, search content and link to published articles, disseminate their full text and use them for any legitimate non-commercial purposes (i.e. educational or scientific) with the mandatory reference to the article’s authors and initial publication in this journal.
- Original published articles cannot be used by users (exept authors) for commercial purposes or distributed by third-party intermediary organizations for a fee.
Deposit Policy
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) during the editorial process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (see this journal’s registered deposit policy at Sherpa/Romeo directory).
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Post-print (post-refereeing manuscript version) and publisher's PDF-version self-archiving is allowed.
- Archiving the pre-print (pre-refereeing manuscript version) not allowed.