Vol. 13 No. 3 (2025)
September 2025

Articles

Understanding artificial intelligence chatbot quality and experience: A higher education student perspective

Published 2025-09-30

  • Ines Dužević
    • ,
  • Tomislav Baković
    • ,
  • Vivien Surman

Ines Dužević
University of Zagreb

Tomislav Baković
University of Zagreb

Vivien Surman
Budapest University of Technology and Economics

Abstract

Objective: The article aims to identify the key aspects that define service quality for artificial intelligence chatbots (AICB) in higher education, based on insights from students. The second objective is to put AICB quality into the broader context of other key variables associated with student experience of AICB, such as AICB adoption, AICB usability, AICB engagement, and AICB mistrust.

Research Design & Methods: Based on extant service quality research and established scale development techniques, the study constructs, refines, and validates a multidimensional AICB service quality scale through a series of pilot and validation studies. The article includes both qualitative and quantitative technics, as we developed a questionnaire based on a literature review and 48 mini focus group interviews. In total, 308 participants filled out the questionnaire. For the analyses, we applied both exploratory and confirmatory factor analysis together with scale validation and correlation analysis.

Findings: We began the AICB service quality scale with 27 items across five dimensions: AICB quality, AICB mistrust, AICB usability, AICB adoption, and AICB engagement. The final scale consisted of 15 items across four dimensions with only AI engagement left out. Data analysis emphasised the critical role of AI quality in AI usability and AI adoption. The research also confirmed AI mistrust is an important aspect with a negative connection to AI quality.

Implications & Recommendations: The study results have several theoretical and practical implications. From the theoretical standpoint, we confirmed that the quality of artificial intelligence (AI) plays a central role in forming student experience. Quality of AICB received the highest score in this analysis (5.03) while AICB mistrust scored lowest (3.58). On the other hand, when it comes to individual correlations between student experience elements and AICB quality, mistrust in AICB shows a negative correlation with the highest score (-0.48). Use and adoption are both connected to AICB quality in a positive way. Results show us there is room for improvement in both AICB quality and student experience since average scores were in the range of 4.5-5.0. The results also emphasised the importance of reducing AICB mistrust for improving AICB quality and overall experience.

Contribution & Value Added: The AICB quality scale facilitates theory development by providing a reliable scale to improve the current understanding of student perceptions regarding different aspects of AICB quality. Higher education institutions (HEI) can use the study results to understand the impact of new technologies such as AICB on student experiences.

Keywords

artificial intelligence (AI), chatbots, artificial intelligence chatbots (AICB), AICB experience, AICB quality, higher education

(PDF) Save

Author Biography

Ines Dužević

Ines Dužević, PhD in Quality Management (2013, Faculty of Economics and Business - Zagreb), Associate professor at the Department of Trade and International Business, Faculty of Economics and Business, University of Zagreb (Croatia). Her research interests include service quality, higher education quality management, and digital transformation of higher education services.

Tomislav Baković

Tomislav Baković, PhD in Quality Management (2010, Faculty of Economics and Business - Zagreb), Full professor at the Department of Trade and International Business, Faculty of Economics and Business, University of Zagreb (Croatia). His research interests include quality management, innovation management, manufacturing industry and commodity markets.

Vivien Surman

Vivien Surman, PhD in Business and Management (2021, Faculty of Economic and Social Sciences – Budapest), Associate professor at the Department of Management and Business Economics, Faculty of Economic and Social Sciences, Budapest University of Technology and Economics (Hungary). Her research interests include service quality, higher education quality, strategic management, and sustainability management.

References

  1. Ali, M., & Abdel-Haq, M.K. (2021). Bibliographical Analysis of Artificial Intelligence Learning in Higher Education: Is the Role of the Human Educator and Educated a Thing of the Past? In Fostering Communication and Learning With Underutilized Technologies in Higher Education, IGI Global. https://doi.org/10.4018/978-1-7998-4846-2.ch003
  2. Al-Mughairi, H., & Bhaskar, P. (2024). Exploring the factors affecting the adoption AI techniques in higher education: insights from teachers’ perspectives on ChatGPT. Journal of Research in Innovative Teaching & Learning, ahead-of-print. https://doi.org/10.1108/JRIT-09-2023-0129
  3. Alwaqdani, M. (2024). Investigating teachers’ perceptions of artificial intelligence tools in education: potential and difficulties. Education and Information Technologies. 30. 2737-2755. https://doi.org/10.1007/s10639-024-12903-9
  4. Al-Zahrani, A.M. (2024). Unveiling the shadows: Beyond the hype of AI in education. Heliyon, 10:e30696. https://doi.org/10.1016/j.heliyon.2024.e30696
  5. Bagozzi, R.P., & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16(1), 74-94.
  6. Bakoban, R.A., & Aljarallah, S.A. (2015). Extracurricular Activities and Their Effect on the Student’s Grade Point Average: Statistical Study. Educational Research and Reviews, 10(20), 2737-2744. https://doi.org/10.5897/ERR2015.2436
  7. Bartholomew, D., Knotts, M., & Moustaki, I. (2011). Latent variable models and factor analysis: A unified approach. (3rd ed.). West Sussex, UK: John Wiley & Sons.
  8. Bedzsula, B., & Tóth, Zs, E. (2019). Identification of critical to service quality attributes in higher education with student involvement. In Proceedings of FEB Zagreb 10th International Odyssey Conference on Economics and Business, Opatia, Crotaia, 317-330.
  9. Blau, I., & Shamir-Inbal, T. (2018). Digital technologies for promoting “student voice” and co-creating learning experience in an academic course. Instructional Sciences, 46(2), 315-336.
  10. Browne, M.W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K.A. Bollen & J.S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage.
  11. Byrne, B.M. (2016). Structural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming (3rd ed.). New York, NY: Routledge.
  12. Chadha, A. (2024). Transforming higher education for the digital age. Journal of Interdisciplinary Studies in Education. https://doi.org/10.32674/em2qsn46
  13. Chen, L., Chen, P., & Lin, Z. (2020). Artificial Intelligence in Education: A Review. IEEE Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510
  14. Close, J. (2018). Tactics and strategies to survive ‘student engagement’, or joining the Soil Society and other stories: A panel discussion. Learning and Teaching: The International Journal of Higher Education in the Social Sciences, 11(1), 109-130.
  15. Cox, A.M. (2021). Exploring the impact of Artificial Intelligence and robots on higher education through literature-based design fictions. International Journal of Educational Technology in Higher Education, 18, 3. https://doi.org/10.1186/s41239-020-00237-8
  16. Crawford, J., Cowling, M., & Allen, K. (2023). Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). Journal of University Teaching and Learning Practice, 20(3). https://doi.org/10.53761/1.20.3.02
  17. Davis, F.D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319-340. https://www.jstor.org/stable/249008
  18. Dean, A., & Gibbs, P. (2015). Student satisfaction or happiness? A preliminary rethink of what is important in the student experience. Quality Assurance in Education, 23, 5-19. https://doi.org/10.1108/QAE-10-2013-0044
  19. Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. (2023). The impact of ChatGPT on higher education. Frontiers in Education, 8, 1-13. https://doi.org/10.3389/feduc.2023.1206936
  20. Dempere, J.M., Modugu, K.P., Hesham, A., & Ramasamy, L.K. (2023). The impact of ChatGPT on higher education. Frontiers in Education. 8:1206936. https://doi.org/10.3389/feduc.2023.1206936
  21. Dodigovic, M. (2007). Artificial intelligence and second language learning: An efficient approach to error remediation. Language Awareness, 16(2), 99-113. https://doi.org/10.2167/la416.0
  22. Dou, R., Zhuang, G., Liu, X., Hou, Y., & Sun, J. (2024). Potential of AI for service performance of manufacturers: Analytical and empirical insights. Advanced Engineering Informatics, 60, 102383. https://doi.org/10.1016/j.aei.2024.102383
  23. Duran, V. (2024). Analyzing teacher candidates’ arguments on AI integration in education via different chatbots. Digital Education Review. https://doi.org/10.1344/der.2024.45.68-83
  24. Dužević, I. (2015). A conceptual framework for analysing the impact of influences on student engagement and learning. Tertiary Education and Management, 21(1), 66-79. https://doi.org/10.1080/13583883.2014.1000368
  25. Dužević, I., Mikulić, J., & Baković, T. (2018). An Extended Framework for Analysing Higher Education Performance. Total Quality Management and Business Excellence, 29(5-6), 599-617. https://doi.org/10.1080/14783363.2016.1224083
  26. Eagle, L., & Brennan, R. (2007). Are students customers? TQM and marketing perspectives. Quality Assurance in Education, 15(1), 44-60.
  27. Elsharnouby, T.H. (2015). Student co-creation behaviour in higher education: the role of satisfaction with the university experience. Journal of Marketing for Higher Education, 25(2), 238-262. https://doi.org/10.1080/08841241.2015.1059919
  28. Field, A. (2009). Discovering Statistics Using SPSS. Sage, London.
  29. Georgescu, A. (2018). Chatbots for education – trends, benefits and challenges. 14th International Conference eLearning and Software for Education. https://doi.org/10.12753/2066-026x-18-097
  30. Goralski, M.A., & Tan, T.K. (2020). Artificial intelligence and sustainable development. The International Journal of Management Education, 18(1), 100330. https://doi.org/10.1016/j.ijme.2019.100330
  31. Grassini, S. (2023). Development and validation of the AI Attitude Scale (AIAS-4): a brief measure of general attitude toward artificial intelligence. Frontiers in Psychology, 14:1191628. https://doi.org/10.3389/fpsyg.2023.1191628
  32. Hamam, D. (2021). The New Teacher Assistant: A Review of Chatbots’ Use in Higher Education. Communications in Computer and Information Science, 1421, 59-63. In 3rd International Conference on Human-Computer Interaction, HCII 2021. https://doi.org/10.1007/978-3-030-78645-8_8
  33. Hannan, E., & Liu, S. (2023). AI: new source of competitiveness in higher education. Competitiveness Review, 33(2), 265-279. https://doi.org/10.1108/CR-03-2021-0045
  34. Heryandi, A. (2020). Developing Chatbot For Academic Record Monitoring in Higher Education Institution. In IOP Conference Series: Materials Science and Engineering, 879. https://doi.org/10.1088/1757-899X/879/1/012049
  35. Hien, H., Cuong, P., Nam, L., Nhung, H., & Dinh, T. (2018). Intelligent Assistants in Higher-Education Environments: The FIT-EBot, a Chatbot for Administrative and Learning Support. In Proceedings of the 9th International Symposium on Information and Communication Technology. https://doi.org/10.1145/3287921.3287937
  36. Hong, S., Park, T., & Choi, J. (2020). Analyzing Research Trends in University Student Experience Based on Topic Modeling. Sustainability, 12(9), 3570. https://doi.org/10.3390/su12093570
  37. Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55.
  38. Huddar, A., Bysani, C., Suchak, C., Kolekar, U., & Upadhyaya, K. (2020). Dexter the college FAQ chatbot. In 2020 International Conference on Convergence to Digital World – Quo Vadis (ICCDW), Mumbai, India, 1-5, https://doi.org/10.1109/ICCDW45521.2020.9318648
  39. Hutson, J., & Plate, D. (2024). The Algorithm of Fear: Unpacking Prejudice Against AI and the Mistrust of Technology. Journal of Innovation and Technology, 2024. Retrieved from https://iuojs.intimal.edu.my/index.php/joit/article/view/625 on June 1, 2023.
  40. Jabborov, A., Kharlamova, A., Kholmatova, Z., Kruglov, A., Kruglov, V., & Succi, G. (2023). “Taxonomy of Quality Assessment for Intelligent Software Systems: A Systematic Literature Review,”. In IEEE Access, 11, 130491-130507. https://doi.org/10.1109/ACCESS.2023.3333920
  41. Kahu, E.R., & Nelson, K. (2018). ‘Hardworking, determined and happy’: First-year students’ understanding and experience of success. Higher Education Research & Development, 37(6), 1260-1273. https://doi.org/10.1080/07294360.2018.1478803
  42. Khare, K., Stewart, B., & Khare, A. (2018). Artificial Intelligence and the Student Experience: An Institutional Perspective. IAFOR Journal of Education, 6(3), https://doi.org/10.22492/ije.6.3.04
  43. Kim-Soon, N., Rahman, A., & Ahmed, M. (2014). E-Service Quality in Higher Education and Frequency of Use of the Service. International Education Studies, 7(3), 1-10., https://doi.org/10.5539/IES.V7N3P1
  44. Kuh, G.D. (1995). The Other Curriculum: Out of class Experiences Associated with Student Learning and Personal Development. The Journal of Higher Education, 66(2), 123-155. https://doi.org/10.1080/00221546.1995.11774770
  45. Lazar, I.M., Panisoara, G., & Panisoara, I.O. (2020). Digital technology adoption scale in the blended learning context in higher education: Development, validation and testing of a specific tool. PLoS ONE, 15(7), e0235957. https://doi.org/10.1371/journal.pone.0235957
  46. Liu, Y., Chen, L., & Yao, Z. (2022). The application of artificial intelligence assistant to deep learning in teachers’ teaching and students’ learning processes. Frontiers in Psychology, 13, 1-13. https://doi.org/10.3389/fpsyg.2022.929175
  47. Lupo, T., & Buscarino, E.A. (2021). Methodological Approach for Developing and Validating a Parsimonious and Robust Measurement Tool: The Academic E-Service Quality (ACEQUAL) Model. Education Sciences, 11(10), 613. https://doi.org/10.3390/educsci11100613
  48. Maeda, T., & Quan-Haase, A. (2024). When Human-AI Interactions Become Parasocial: Agency and Anthropomorphism in Affective Design. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3630106.3658956
  49. Marcus, A.M., Ovinova, L.N., Dmitrusenko, I.N., & Shraiber, E.G. (2023). Application of Artificial Intelligence Technology in Teaching English Language to Engineering Bachelors. In 2023 International Conference on Quality Management, Transport and Information Security, Information Technologies (IT&QM&IS), Petrozavodsk, Russian Federation, 147-151. https://doi.org/10.1109/ITQMTIS58985.2023.10346594
  50. Marimon, F., Mas-Machuca, M., & Akhmedova, A. (2024). Trusting in Generative AI: Catalyst for Employee Performance and Engagement in the Workplace. International Journal of Human–Computer Interaction, 1-16. https://doi.org/10.1080/10447318.2024.2388482
  51. Matus, N., Rusu, C., & Cano, S. (2021). Student eXperience: A Systematic Literature Review. Applied Sciences, 11(20), 9543. https://doi.org/10.3390/app11209543
  52. McDonald, R.P. (1985). Factor analysis and related methods. Hillside, NJ: Lawrence Erlbaum Associates, Inc.
  53. Mollick, E. (2024). Co-inteligence: Living and working with AI. London: Penguin Random House.
  54. Neumann, M., Rauschenberger, M., & Schön, E. (2023). “We Need To Talk About ChatGPT”: The Future of AI and Higher Education. In 2023 IEEE/ACM 5th International Workshop on Software Engineering Education for the Next Generation (SEENG), 29-32. https://doi.org/10.1109/SEENG59157.2023.00010
  55. Niemi, H. (2021). AI in learning: Preparing ground for future learning. Journal of Pacific Rim Psychology, 15, 1-12. https://doi.org/10.1177/18344909211038105
  56. Noor, N., Rao Hill, S., & Troshani, I. (2022). Developing a service quality scale for artificial intelligence service agents. European Journal of Marketing, 56(5), 1301-1336. https://doi.org/10.1108/EJM-09-2020-0672
  57. Parasuraman, A., Zeithaml, V.A., & Malhotra, A. (2005). E-S-QUAL A Multiple-Item Scale for Assessing Electronic Service Quality. Journal of Service Research, 7(3), 213-233; https://doi.org/10.1177/1094670504271156
  58. Pereira, D.S.M., Falcão, F., Costa, L., Lunn, B.S., Pêgo, J.M., & Costa, P. (2023). Here’s to the future: Conversational agents in higher education- a scoping review. International Journal of Educational Research, 122, 102233. https://doi.org/10.1016/j.ijer.2023.102233
  59. Phogat, S., & Gupta, A.M. (2018). Evaluating the elements of just in time (JIT) for implementation in maintenance by exploratory and confirmatory factor analysis. International Journal of Quality & Reliability Management, 36(1), 7-24. https://doi.org/10.1108/IJQRM-12-2017-0279
  60. Poole, D.L., & Mackworth, A.K. (2010). Artificial Intelligence: foundations of computational agents. Cambridge: Cambridge University Press.
  61. Prentice, C. (2023). Developing a Scale to Measure Artificial Intelligence Service Quality. In Leveraging Emotional and Artificial Intelligence for Organisational Performance. Springer, Singapore. https://doi.org/10.1007/978-981-99-1865-2_8
  62. Qian, C., Yeming, G., Yaobin, L., & Jing, T. (2022). Classifying and measuring the service quality of AI chatbot in frontline service. Journal of Business Research, 145, 552-568. https://doi.org/10.1016/j.jbusres.2022.02.088
  63. Rudolph, J., Tan, S., & Tan, S. (2023). War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. Journal of Applied Learning & Teaching, 6(1), 364-389. https://doi.org/10.37074/jalt.2023.6.1.23
  64. Scharowski, N., Perrig, S.A.C., Aeschbach, L.F., von Felten, N., Opwis, K., Wintersberger, P., & Brühlmann, F. (2024). To Trust or Distrust Trust Measures: Validating Questionnaires for Trust in AI (2024). CoRR, 2403.00582. https://doi.org/10.48550/ARXIV.2403.00582
  65. Siau, K. (2017). Impact of Artificial Intelligence, Robotics, and Automation on Higher Education. AMCIS 2017 Proceedings, 63. Retrieved from https://aisel.aisnet.org/amcis2017/TREOs/Presentations/63 on September, 20, 2024.
  66. Slimi, Z., & Carballido, B.V. (2023). Systematic Review AI’s Impact on Higher Education – Learning, Teaching and Career Opportunities. TEM Journal, 12(3), 1627-1637. https://doi.org/10.18421/TEM123-44
  67. Stiwne, E.E., & Jungert, T. (2010). Engineering students’ experiences of transition from study to work. Journal of Education and Work, 23(5), 417-437. https://doi.org/10.1080/13639080.2010.515967
  68. Surman, V., Tóth, Zs.E., & Danó, Gy. (2022). Defining Service Quality Attributes at Different Levels of Operation in Higher Education Institutions: How Could the Service Quality Perceptions of Students Contribute to a Better Understanding of Improvement Directions?. Zagreb International Review of Economics and Business, S1, 1-15. https://doi.org/10.2478/zireb-2022-0021
  69. Tan, AH.T., Muskat, B., & Zehrer, A. (2016). A Systematic Review of Quality of Student Experience in Higher Education. International Journal of Quality and Service Sciences, 8(2), 209-228. https://doi.org/10.1108/IJQSS-08-2015-0058
  70. Tight, M. (2012). Researching Higher Education. McGraw-Hill Education: Maidenhead, UK.
  71. Tóth, Zs., & Surman, V. (2019). Listening to the voice of students, developing a service quality measuring and evaluating framework for a special course. International Journal of Quality and Service Sciences, 11(4), 455-472. https://doi.org/10.1108/IJQSS-02-2019-0025
  72. Wang, T., Lund, B.D., Marengo, A., Pagano, A., Mannuru, N.R., Teel, Z., & Pange, J. (2023). Exploring the Potential Impact of Artificial Intelligence (AI) on International Students in Higher Education: Generative AI, Chatbots, Analytics, and International Student Success. Applied Sciences, 13(11), 6716. https://doi.org/10.3390/app13116716
  73. Weichbroth, P. (2020). Usability of Mobile Applications: A Systematic Literature Study. IEEE Access, 8, 55563-55577. https://doi.org/10.1109/ACCESS.2020.2981892
  74. Westman, S., Kauttonen, J., Klemetti, A., Korhonen, N., Manninen, M., Mononen, A., Niittymaki, S., & Paananen, H. (2021). Artificial Intelligence for Career Guidance – Current Requirements and Prospects for the Future, IAFOR Journal of Education, 9(4), 43-62. https://doi.org/10.22492/ije.9.4.03
  75. Williams, R.T. (2024). The ethical implications of using generative chatbots in higher education. Frontiers in Education, 8, 1331607. https://doi.org/10.3389/feduc.2023.1331607
  76. Yan, T., Zhang, Y., & Hua, R. (2023). Artificial Intelligence Enterprise Service Quality Evaluation System. In 2023 Asia Conference on Advanced Robotics, Automation, and Control Engineering (ARACE), Chengdu, China, 23-28. https://doi.org/10.1109/ARACE60380.2023.00012
  77. Yang, S., & Evans, C. (2019). Opportunities and Challenges in Using AI Chatbots in Higher Education. Proceedings of the 2019 3rd International Conference on Education and E-Learning. https://doi.org/10.1145/3371647.3371659
  78. Yong, A.G., & Pearce, S. (2013). A Beginner’s Guide to Factor Analysis: Focusing on Exploratory Factor Analysis. Tutorials in Quantitative Methods for Psychology, 9(2), 79-94. https://doi.org/10.20982/TQMP.09.2.P079

Downloads

Download data is not yet available.