A multi-criterion model for evaluating the quality of qualitative research

Published 2023-11-13

  • Rômulo Andrade de Souza Neto
    • ,
  • Daniel Francisco Dumaresq de Souza
    • ,
  • Gabriela Figueiredo Dias
    • ,
  • Maxwell dos Santos Celestino
    • ,
  • André Morais Gurgel
    • ,
  • Anatália Saraiva Martins Ramos
    • ,
  • Marcelo Victor Alves Bila Queiroz
    • ,
  • Joyce Mariella Medeiros Cavalcanti
    • ,
  • Alice Gerlane Cardoso da Silva
PDF

Keywords: research evaluation, qualitative research, quality criteria, multi-criterion methods, ELECTRE TRI

Abstract

The aim of this technological article is to propose a multi-criterion model to evaluate the quality of qualitative research. The justification for the work is the growing number of qualitative researches and the scarcity of objetive evaluation methods for this kind of research. The theoretical foundation is the quality literature of qualitative research. The method employed is ELECTRE TRI because it meets the requirements of the problem analyzed, such as multiple criteria and problematic classification. In modeling, seven criteria (theoretical consistency, relevance of the search, data fidelity, transferability of the results, credibility of the results, textual and ethical clarity) and three classes (accepted, revise and resubmit and rejected) were defined. The validation of the model was carried out with a specialist in qualitative methods, who also defined the test parameters. The model was tested with ten qualitative articles published in annals of a national scientific event following the parameters pre-established by the specialist. The results show that the model ranked the ten articles among the three predefined classes, with only one article reaching the "approved" class. The model can be used by scientific journals to standardize the process of evaluating qualitative research and to help the preparation of opinions.

References

  1. ALMEIDA, A. T. Decision process in organizations: building multi-criteria decision models. São Paulo: Atlas, 2013.
  2. ANPAD - NATIONAL ASSOCIATION OF GRADUATE STUDIES AND RESEARCH IN ADMINISTRATION. Events. Available at: < https://anpad.com.br/pt_br/index_event>. Accessed: 08 Sep. 2023.
  3. BECK, C. Qualitative research: the evaluation of its credibility, fittingness, and auditability. Western Journal of Nursing Research, v. 15, n. 2, pp. 263-266, 1993. DOI: https://doi.org/10.1177/019394599301500212.
  4. BLAXTER, M. Criteria for the evaluation of qualitative research papers Medical Sociology News, v. 22, n. 1, pp. 68-71, 1996.
  5. BEYER, J. M.; CHANOVE, R. G.; FOX, W. B. The review process and the fates of manuscripts submitted to AMJ. Academy of Management Journal, v. 38, n. 5, p. 1219-1260, 1995. DOI: https://doi.org/10.2307/256856.
  6. CORREA, F. P. The evaluation of qualitative research: a reflection from a justice perspective. Qualitative inquiry, v. 19, para. 3, p. 209-218, 2013. DOI: https://doi.org/10.1177/1077800412466225.
  7. COSTA, H. G.; MANSUR, A. F. U.; FREITAS, A. L. P.; CARVALHO, R. A. ELECTRE TRI applied the consumer satisfaction assessment. Production, v. 17, para. 2, pp. 230-245, 2007. DOI: https://doi.org/10.1590/S0103-65132007000200002.
  8. CRESWELL, J. W. Educational research: planning, conducting, and evaluating quantitative and qualitative research (4th ed.). Boston: Pearson Education, 2012.
  9. CRESWELL, J. W. Qualitative research and research project: choosing from five approaches, 3 ed. Porto Alegre: Pensao, 2014a.
  10. CRESWELL, J. W. Research design: qualitative, quantitative, and mixed methods approaches, 4 ed. Thousand Oaks, CA: Sage, 2014b.
  11. DENZIN, N. K.; LINCOLN, Y. S. Introduction: The Discipline and Practice of Qualitative Research. IN: DENZIN, N. K.; LINCOLN, Y. S (Eds.). The SAGE handbook of qualitative research. 5th ed. Thousand Oaks, CA: Sage, 2018. p. 29-71.
  12. ELLIOTT, R.; FISCHER, C. T.; RENNIE, D. L. Evolving guidelines for publication of qualitative research studies in psychology and related fields. British Journal of Clinical Psychology, v. 38, n. 3, pp. 215-229, 1999. DOI: https://doi.org/10.1348/014466599162782.
  13. FLICK, U. Managing quality in qualitative research. London: Sage, 2007.
  14. FLICK, U. Introduction to qualitative research. 3 ed. Porto Alegre: Artmed, 2009.
  15. FOSSEY E.; HARVEY C.; MCDERMOTT F.; DAVIDSON L. Understanding and evaluating qualitative research. Australian and New Zealand Journal of Psychiatry, v. 36, n. 6, pp. 717-732, 2002. DOI: https://doi.org/10.1046/j.1440-1614.2002.01100.x.
  16. GIBBS, G. R. Analysis of qualitative data. São Paulo: Artmed, 2009.
  17. HWANG, C. L.; YOON, K. Multiple attribute decision making: methods and applications a state-of-the-art survey. Berlin: Springer-Verlag, 1981.
  18. IJDAR. Guidelines for associate editors. Available í: <http://www.springer.com/-computer/image+processing/journal/10032>. Accessed: 22 Dec. 2016.
  19. KITTO, S.; CHESTERS, J.; GRBICH, C. Quality in qualitative research: criteria for authors and assessor in the submission and assessment of qualitative research articles for the medical journal of Australia. Medical Journal of Australia, v. 188, no. 4, pp. 243-246, 2008.
  20. KREFTING, L. Rigor in qualitative research: the assessment of trustworthiness. The American Journal of Occupational Therapy, v. 45, n. 3, pp. 214-222, 1991. DOI: https://doi.org/10.5014/ajot.45.3.214.
  21. KUPER A.; LINGARD L.; LEVINSON, W. Critically appraising qualitative research. British Medical Journal-BMJ, v. 337, n. a1035, p. 687-689, 2008. DOI: https://doi.org/10.1136/bmj.a1035.
  22. LINCOLN, Y. S.; GUBA, E. A. Naturalistic Inquiry. Beverly Hills, CA: Sage, 1985.
  23. GODFREY, M. An evaluation tool to assess the quality of qualitative research studies. International Journal Social Research Methodology, v. 7, n. 2, pp. 181-196, 2004. DOI: https://doi.org/10.1080/1364557032000045302.
  24. LOSEKE, D. R.; CAHILL, S. E. Publishing qualitative manuscripts: lessons learned. In: SEALE, C.; GOBO, G.; GUBRIUM, J. F.; SILVERMAN, D. (Eds.). Qualitative research practice. London: Sage, 2007. p. 491-506.
  25. MERRIAM, S. B.; TISDELL, E. J. Qualitative research: a guide to design and implementation. 4ed. San Francisco: Jossey Bass, 2016.
  26. MILES, M.; HUBERMAN, A.; SALDAÑA, J. Qualitative data analysis: a methods sourcebook. 3 ed. Los Angeles: Sage, 2014.
  27. MORSE, J. Reframing rigor in qualitative inquiry. IN: DENZIN, N. K.; LINCOLN, Y. S (Eds.). The SAGE handbook of qualitative research. 5 ed. Thousand Oaks, CA: Sage, 2018. p. 1373-1409.
  28. MOUSSEAU, V.; SLOWINSKI, R.; ZIELNIEWICZ, P. A user-oriented implementation of the ELECTRE-TRI method integrating preference elicitation support. Computers & Operations Research, v. 27, 7-8, p. 757-777, 2000. DOI: https://doi.org/10.1016/S0305-0548(99)00117-3.
  29. O’BRIEN, B. C.; HARRIS, I. B.; BECKMAN, T. J.; REED, D. A.; COOK, D. A. Standards for reporting qualitative research: a synthesis of recommendations. Academic Medicine, v. 89, para. 9, p. 1245-1251, 2014. DOI: https://doi.org/10.1097/ACM.0000000000000388.
  30. RAGIN, C.; NAGEL, J.; WHITE, P. Workshop on Scientific Foundations of Qualitative Research. Washington, DC: National Science Foundation, 2004. Available at: <http://www.nsf.gov/pubs/2004/nsf04219/nsf04219.pdf>. Accessed 08 Sep. 2023.
  31. REID A.; GOUGH, S. Guidelines for reporting and evaluating qualitative research: what are the alternatives. Environmental educational research, v. 6, n. 1, pp. 59-91, 2000. DOI: https://doi.org/10.1080/135046200110494.
  32. RICHARDSON, L.; ST. PIERRE, E. A. Writing: A method of inquiry. IN: DENZIN, N. K.; LINCOLN, Y. S (Eds.). The SAGE handbook of qualitative research. 5 ed. Thousand Oaks, CA: Sage, 2018. p. 1410-1444
  33. ROY, B. Multicriteria methodology for decision aiding. Dordrecht: Kluwer Academic Publishers, 1996. DOI: https://doi.org/10.1007/978-1-4757-2500-1.
  34. RUSSEL, C.; GREGORY, D. Evaluation of qualitative research studies. Evidence-Based Nursing, v. 6, n. 2, pp. 36-40, 2003. DOI: https://doi.org/10.1136/ebn.6.2.36.
  35. SILVERMAN, D.; MARVASTI, A. Doing qualitative research: a comprehensive guide. Los Angeles: SAGE Publications, 2008.
  36. SOUZA, R. A.; DIAS, G. F.; SILVA, R. R.; RAMOS, A. S. M. Effects of qualitative data analysis software on research quality. Journal of Contemporary Administration, v. 23, n. 3, p. 373-394, 2019. DOI: https://doi.org/10.1590/1982-7849rac2019170357.
  37. SPENCER, L.; RITCHIE, J.; LEWIS, J.; DILLON, L. Quality in qualitative evaluation: a framework for assessing research evidence. London: Crown, 2003.
  38. Evidence, criteria, policy, and politics: the debate about quality and utility in educational and social research. IN: DENZIN, N. K.; LINCOLN, Y. S (Eds.). The SAGE handbook of qualitative research. 5 ed. Thousand Oaks, CA: Sage, 2018. p. 1320-1372.
  39. TRACY, S. J. Qualitative research methods: collecting evidence, crafting analysis, communicating impact. West Sussex: Wiley-Blackwell, 2013.
  40. WHITTEMORE, R.; CHASE, S. K.; MANDLE, C. L. Validity in qualitative research. Qualitative health research, v. 11, para. 4, pp. 522-537, 2001. DOI: https://doi.org/10.1177/104973201129119299.
  41. YIN, R. K. Qualitative research: from start to finish. New York: The Guilford Press, 2011.

How to Cite

de Souza Neto, R. A., de Souza, D. F. D., Dias, G. F., Celestino, M. dos S., Gurgel, A. M., Ramos, A. S. M., Queiroz, M. V. A. B., Cavalcanti, J. M. M., & da Silva, A. G. C. (2023). A multi-criterion model for evaluating the quality of qualitative research. International Journal of Scientific Management and Tourism, 9(6), 3884–3911. https://doi.org/10.55905/ijsmtv9n6-030

Download Citation

Current Issue

MOST READ LAST WEEK
Keywords

Language

Information