Understanding artificial intelligence chatbot quality and experience: A higher education student perspective

Abstract
Objective: The article aims to identify the key aspects that define service quality for artificial intelligence chatbots (AICB) in higher education, based on insights from students. The second objective is to put AICB quality into the broader context of other key variables associated with student experience of AICB, such as AICB adoption, AICB usability, AICB engagement, and AICB mistrust.
Research Design & Methods: Based on extant service quality research and established scale development techniques, the study constructs, refines, and validates a multidimensional AICB service quality scale through a series of pilot and validation studies. The article includes both qualitative and quantitative technics, as we developed a questionnaire based on a literature review and 48 mini focus group interviews. In total, 308 participants filled out the questionnaire. For the analyses, we applied both exploratory and confirmatory factor analysis together with scale validation and correlation analysis.
Findings: We began the AICB service quality scale with 27 items across five dimensions: AICB quality, AICB mistrust, AICB usability, AICB adoption, and AICB engagement. The final scale consisted of 15 items across four dimensions with only AI engagement left out. Data analysis emphasised the critical role of AI quality in AI usability and AI adoption. The research also confirmed AI mistrust is an important aspect with a negative connection to AI quality.
Implications & Recommendations: The study results have several theoretical and practical implications. From the theoretical standpoint, we confirmed that the quality of artificial intelligence (AI) plays a central role in forming student experience. Quality of AICB received the highest score in this analysis (5.03) while AICB mistrust scored lowest (3.58). On the other hand, when it comes to individual correlations between student experience elements and AICB quality, mistrust in AICB shows a negative correlation with the highest score (-0.48). Use and adoption are both connected to AICB quality in a positive way. Results show us there is room for improvement in both AICB quality and student experience since average scores were in the range of 4.5-5.0. The results also emphasised the importance of reducing AICB mistrust for improving AICB quality and overall experience.
Contribution & Value Added: The AICB quality scale facilitates theory development by providing a reliable scale to improve the current understanding of student perceptions regarding different aspects of AICB quality. Higher education institutions (HEI) can use the study results to understand the impact of new technologies such as AICB on student experiences.
Keywords
artificial intelligence (AI), chatbots, artificial intelligence chatbots (AICB), AICB experience, AICB quality, higher education
Author Biography
Ines Dužević
Ines Dužević, PhD in Quality Management (2013, Faculty of Economics and Business - Zagreb), Associate professor at the Department of Trade and International Business, Faculty of Economics and Business, University of Zagreb (Croatia). Her research interests include service quality, higher education quality management, and digital transformation of higher education services.
Tomislav Baković
Tomislav Baković, PhD in Quality Management (2010, Faculty of Economics and Business - Zagreb), Full professor at the Department of Trade and International Business, Faculty of Economics and Business, University of Zagreb (Croatia). His research interests include quality management, innovation management, manufacturing industry and commodity markets.
Vivien Surman
Vivien Surman, PhD in Business and Management (2021, Faculty of Economic and Social Sciences – Budapest), Associate professor at the Department of Management and Business Economics, Faculty of Economic and Social Sciences, Budapest University of Technology and Economics (Hungary). Her research interests include service quality, higher education quality, strategic management, and sustainability management.
References
- Ali, M., & Abdel-Haq, M.K. (2021). Bibliographical Analysis of Artificial Intelligence Learning in Higher Education: Is the Role of the Human Educator and Educated a Thing of the Past? In Fostering Communication and Learning With Underutilized Technologies in Higher Education, IGI Global. https://doi.org/10.4018/978-1-7998-4846-2.ch003
- Al-Mughairi, H., & Bhaskar, P. (2024). Exploring the factors affecting the adoption AI techniques in higher education: insights from teachers’ perspectives on ChatGPT. Journal of Research in Innovative Teaching & Learning, ahead-of-print. https://doi.org/10.1108/JRIT-09-2023-0129
- Alwaqdani, M. (2024). Investigating teachers’ perceptions of artificial intelligence tools in education: potential and difficulties. Education and Information Technologies. 30. 2737-2755. https://doi.org/10.1007/s10639-024-12903-9
- Al-Zahrani, A.M. (2024). Unveiling the shadows: Beyond the hype of AI in education. Heliyon, 10:e30696. https://doi.org/10.1016/j.heliyon.2024.e30696
- Bagozzi, R.P., & Yi, Y. (1988). On the evaluation of structural equation models. Journal of the Academy of Marketing Science, 16(1), 74-94.
- Bakoban, R.A., & Aljarallah, S.A. (2015). Extracurricular Activities and Their Effect on the Student’s Grade Point Average: Statistical Study. Educational Research and Reviews, 10(20), 2737-2744. https://doi.org/10.5897/ERR2015.2436
- Bartholomew, D., Knotts, M., & Moustaki, I. (2011). Latent variable models and factor analysis: A unified approach. (3rd ed.). West Sussex, UK: John Wiley & Sons.
- Bedzsula, B., & Tóth, Zs, E. (2019). Identification of critical to service quality attributes in higher education with student involvement. In Proceedings of FEB Zagreb 10th International Odyssey Conference on Economics and Business, Opatia, Crotaia, 317-330.
- Blau, I., & Shamir-Inbal, T. (2018). Digital technologies for promoting “student voice” and co-creating learning experience in an academic course. Instructional Sciences, 46(2), 315-336.
- Browne, M.W., & Cudeck, R. (1993). Alternative ways of assessing model fit. In K.A. Bollen & J.S. Long (Eds.), Testing structural equation models (pp. 136-162). Newbury Park, CA: Sage.
- Byrne, B.M. (2016). Structural Equation Modeling with AMOS: Basic Concepts, Applications, and Programming (3rd ed.). New York, NY: Routledge.
- Chadha, A. (2024). Transforming higher education for the digital age. Journal of Interdisciplinary Studies in Education. https://doi.org/10.32674/em2qsn46
- Chen, L., Chen, P., & Lin, Z. (2020). Artificial Intelligence in Education: A Review. IEEE Access, 8, 75264-75278. https://doi.org/10.1109/ACCESS.2020.2988510
- Close, J. (2018). Tactics and strategies to survive ‘student engagement’, or joining the Soil Society and other stories: A panel discussion. Learning and Teaching: The International Journal of Higher Education in the Social Sciences, 11(1), 109-130.
- Cox, A.M. (2021). Exploring the impact of Artificial Intelligence and robots on higher education through literature-based design fictions. International Journal of Educational Technology in Higher Education, 18, 3. https://doi.org/10.1186/s41239-020-00237-8
- Crawford, J., Cowling, M., & Allen, K. (2023). Leadership is needed for ethical ChatGPT: Character, assessment, and learning using artificial intelligence (AI). Journal of University Teaching and Learning Practice, 20(3). https://doi.org/10.53761/1.20.3.02
- Davis, F.D. (1989). Perceived Usefulness, Perceived Ease of Use, and User Acceptance of Information Technology. MIS Quarterly, 13(3), 319-340. https://www.jstor.org/stable/249008
- Dean, A., & Gibbs, P. (2015). Student satisfaction or happiness? A preliminary rethink of what is important in the student experience. Quality Assurance in Education, 23, 5-19. https://doi.org/10.1108/QAE-10-2013-0044
- Dempere, J., Modugu, K., Hesham, A., & Ramasamy, L. (2023). The impact of ChatGPT on higher education. Frontiers in Education, 8, 1-13. https://doi.org/10.3389/feduc.2023.1206936
- Dempere, J.M., Modugu, K.P., Hesham, A., & Ramasamy, L.K. (2023). The impact of ChatGPT on higher education. Frontiers in Education. 8:1206936. https://doi.org/10.3389/feduc.2023.1206936
- Dodigovic, M. (2007). Artificial intelligence and second language learning: An efficient approach to error remediation. Language Awareness, 16(2), 99-113. https://doi.org/10.2167/la416.0
- Dou, R., Zhuang, G., Liu, X., Hou, Y., & Sun, J. (2024). Potential of AI for service performance of manufacturers: Analytical and empirical insights. Advanced Engineering Informatics, 60, 102383. https://doi.org/10.1016/j.aei.2024.102383
- Duran, V. (2024). Analyzing teacher candidates’ arguments on AI integration in education via different chatbots. Digital Education Review. https://doi.org/10.1344/der.2024.45.68-83
- Dužević, I. (2015). A conceptual framework for analysing the impact of influences on student engagement and learning. Tertiary Education and Management, 21(1), 66-79. https://doi.org/10.1080/13583883.2014.1000368
- Dužević, I., Mikulić, J., & Baković, T. (2018). An Extended Framework for Analysing Higher Education Performance. Total Quality Management and Business Excellence, 29(5-6), 599-617. https://doi.org/10.1080/14783363.2016.1224083
- Eagle, L., & Brennan, R. (2007). Are students customers? TQM and marketing perspectives. Quality Assurance in Education, 15(1), 44-60.
- Elsharnouby, T.H. (2015). Student co-creation behaviour in higher education: the role of satisfaction with the university experience. Journal of Marketing for Higher Education, 25(2), 238-262. https://doi.org/10.1080/08841241.2015.1059919
- Field, A. (2009). Discovering Statistics Using SPSS. Sage, London.
- Georgescu, A. (2018). Chatbots for education – trends, benefits and challenges. 14th International Conference eLearning and Software for Education. https://doi.org/10.12753/2066-026x-18-097
- Goralski, M.A., & Tan, T.K. (2020). Artificial intelligence and sustainable development. The International Journal of Management Education, 18(1), 100330. https://doi.org/10.1016/j.ijme.2019.100330
- Grassini, S. (2023). Development and validation of the AI Attitude Scale (AIAS-4): a brief measure of general attitude toward artificial intelligence. Frontiers in Psychology, 14:1191628. https://doi.org/10.3389/fpsyg.2023.1191628
- Hamam, D. (2021). The New Teacher Assistant: A Review of Chatbots’ Use in Higher Education. Communications in Computer and Information Science, 1421, 59-63. In 3rd International Conference on Human-Computer Interaction, HCII 2021. https://doi.org/10.1007/978-3-030-78645-8_8
- Hannan, E., & Liu, S. (2023). AI: new source of competitiveness in higher education. Competitiveness Review, 33(2), 265-279. https://doi.org/10.1108/CR-03-2021-0045
- Heryandi, A. (2020). Developing Chatbot For Academic Record Monitoring in Higher Education Institution. In IOP Conference Series: Materials Science and Engineering, 879. https://doi.org/10.1088/1757-899X/879/1/012049
- Hien, H., Cuong, P., Nam, L., Nhung, H., & Dinh, T. (2018). Intelligent Assistants in Higher-Education Environments: The FIT-EBot, a Chatbot for Administrative and Learning Support. In Proceedings of the 9th International Symposium on Information and Communication Technology. https://doi.org/10.1145/3287921.3287937
- Hong, S., Park, T., & Choi, J. (2020). Analyzing Research Trends in University Student Experience Based on Topic Modeling. Sustainability, 12(9), 3570. https://doi.org/10.3390/su12093570
- Hu, L., & Bentler, P.M. (1999). Cutoff criteria for fit indexes in covariance structure analysis: Conventional criteria versus new alternatives. Structural Equation Modeling: A Multidisciplinary Journal, 6(1), 1-55.
- Huddar, A., Bysani, C., Suchak, C., Kolekar, U., & Upadhyaya, K. (2020). Dexter the college FAQ chatbot. In 2020 International Conference on Convergence to Digital World – Quo Vadis (ICCDW), Mumbai, India, 1-5, https://doi.org/10.1109/ICCDW45521.2020.9318648
- Hutson, J., & Plate, D. (2024). The Algorithm of Fear: Unpacking Prejudice Against AI and the Mistrust of Technology. Journal of Innovation and Technology, 2024. Retrieved from https://iuojs.intimal.edu.my/index.php/joit/article/view/625 on June 1, 2023.
- Jabborov, A., Kharlamova, A., Kholmatova, Z., Kruglov, A., Kruglov, V., & Succi, G. (2023). “Taxonomy of Quality Assessment for Intelligent Software Systems: A Systematic Literature Review,”. In IEEE Access, 11, 130491-130507. https://doi.org/10.1109/ACCESS.2023.3333920
- Kahu, E.R., & Nelson, K. (2018). ‘Hardworking, determined and happy’: First-year students’ understanding and experience of success. Higher Education Research & Development, 37(6), 1260-1273. https://doi.org/10.1080/07294360.2018.1478803
- Khare, K., Stewart, B., & Khare, A. (2018). Artificial Intelligence and the Student Experience: An Institutional Perspective. IAFOR Journal of Education, 6(3), https://doi.org/10.22492/ije.6.3.04
- Kim-Soon, N., Rahman, A., & Ahmed, M. (2014). E-Service Quality in Higher Education and Frequency of Use of the Service. International Education Studies, 7(3), 1-10., https://doi.org/10.5539/IES.V7N3P1
- Kuh, G.D. (1995). The Other Curriculum: Out of class Experiences Associated with Student Learning and Personal Development. The Journal of Higher Education, 66(2), 123-155. https://doi.org/10.1080/00221546.1995.11774770
- Lazar, I.M., Panisoara, G., & Panisoara, I.O. (2020). Digital technology adoption scale in the blended learning context in higher education: Development, validation and testing of a specific tool. PLoS ONE, 15(7), e0235957. https://doi.org/10.1371/journal.pone.0235957
- Liu, Y., Chen, L., & Yao, Z. (2022). The application of artificial intelligence assistant to deep learning in teachers’ teaching and students’ learning processes. Frontiers in Psychology, 13, 1-13. https://doi.org/10.3389/fpsyg.2022.929175
- Lupo, T., & Buscarino, E.A. (2021). Methodological Approach for Developing and Validating a Parsimonious and Robust Measurement Tool: The Academic E-Service Quality (ACEQUAL) Model. Education Sciences, 11(10), 613. https://doi.org/10.3390/educsci11100613
- Maeda, T., & Quan-Haase, A. (2024). When Human-AI Interactions Become Parasocial: Agency and Anthropomorphism in Affective Design. Proceedings of the 2024 ACM Conference on Fairness, Accountability, and Transparency. https://doi.org/10.1145/3630106.3658956
- Marcus, A.M., Ovinova, L.N., Dmitrusenko, I.N., & Shraiber, E.G. (2023). Application of Artificial Intelligence Technology in Teaching English Language to Engineering Bachelors. In 2023 International Conference on Quality Management, Transport and Information Security, Information Technologies (IT&QM&IS), Petrozavodsk, Russian Federation, 147-151. https://doi.org/10.1109/ITQMTIS58985.2023.10346594
- Marimon, F., Mas-Machuca, M., & Akhmedova, A. (2024). Trusting in Generative AI: Catalyst for Employee Performance and Engagement in the Workplace. International Journal of Human–Computer Interaction, 1-16. https://doi.org/10.1080/10447318.2024.2388482
- Matus, N., Rusu, C., & Cano, S. (2021). Student eXperience: A Systematic Literature Review. Applied Sciences, 11(20), 9543. https://doi.org/10.3390/app11209543
- McDonald, R.P. (1985). Factor analysis and related methods. Hillside, NJ: Lawrence Erlbaum Associates, Inc.
- Mollick, E. (2024). Co-inteligence: Living and working with AI. London: Penguin Random House.
- Neumann, M., Rauschenberger, M., & Schön, E. (2023). “We Need To Talk About ChatGPT”: The Future of AI and Higher Education. In 2023 IEEE/ACM 5th International Workshop on Software Engineering Education for the Next Generation (SEENG), 29-32. https://doi.org/10.1109/SEENG59157.2023.00010
- Niemi, H. (2021). AI in learning: Preparing ground for future learning. Journal of Pacific Rim Psychology, 15, 1-12. https://doi.org/10.1177/18344909211038105
- Noor, N., Rao Hill, S., & Troshani, I. (2022). Developing a service quality scale for artificial intelligence service agents. European Journal of Marketing, 56(5), 1301-1336. https://doi.org/10.1108/EJM-09-2020-0672
- Parasuraman, A., Zeithaml, V.A., & Malhotra, A. (2005). E-S-QUAL A Multiple-Item Scale for Assessing Electronic Service Quality. Journal of Service Research, 7(3), 213-233; https://doi.org/10.1177/1094670504271156
- Pereira, D.S.M., Falcão, F., Costa, L., Lunn, B.S., Pêgo, J.M., & Costa, P. (2023). Here’s to the future: Conversational agents in higher education- a scoping review. International Journal of Educational Research, 122, 102233. https://doi.org/10.1016/j.ijer.2023.102233
- Phogat, S., & Gupta, A.M. (2018). Evaluating the elements of just in time (JIT) for implementation in maintenance by exploratory and confirmatory factor analysis. International Journal of Quality & Reliability Management, 36(1), 7-24. https://doi.org/10.1108/IJQRM-12-2017-0279
- Poole, D.L., & Mackworth, A.K. (2010). Artificial Intelligence: foundations of computational agents. Cambridge: Cambridge University Press.
- Prentice, C. (2023). Developing a Scale to Measure Artificial Intelligence Service Quality. In Leveraging Emotional and Artificial Intelligence for Organisational Performance. Springer, Singapore. https://doi.org/10.1007/978-981-99-1865-2_8
- Qian, C., Yeming, G., Yaobin, L., & Jing, T. (2022). Classifying and measuring the service quality of AI chatbot in frontline service. Journal of Business Research, 145, 552-568. https://doi.org/10.1016/j.jbusres.2022.02.088
- Rudolph, J., Tan, S., & Tan, S. (2023). War of the chatbots: Bard, Bing Chat, ChatGPT, Ernie and beyond. The new AI gold rush and its impact on higher education. Journal of Applied Learning & Teaching, 6(1), 364-389. https://doi.org/10.37074/jalt.2023.6.1.23
- Scharowski, N., Perrig, S.A.C., Aeschbach, L.F., von Felten, N., Opwis, K., Wintersberger, P., & Brühlmann, F. (2024). To Trust or Distrust Trust Measures: Validating Questionnaires for Trust in AI (2024). CoRR, 2403.00582. https://doi.org/10.48550/ARXIV.2403.00582
- Siau, K. (2017). Impact of Artificial Intelligence, Robotics, and Automation on Higher Education. AMCIS 2017 Proceedings, 63. Retrieved from https://aisel.aisnet.org/amcis2017/TREOs/Presentations/63 on September, 20, 2024.
- Slimi, Z., & Carballido, B.V. (2023). Systematic Review AI’s Impact on Higher Education – Learning, Teaching and Career Opportunities. TEM Journal, 12(3), 1627-1637. https://doi.org/10.18421/TEM123-44
- Stiwne, E.E., & Jungert, T. (2010). Engineering students’ experiences of transition from study to work. Journal of Education and Work, 23(5), 417-437. https://doi.org/10.1080/13639080.2010.515967
- Surman, V., Tóth, Zs.E., & Danó, Gy. (2022). Defining Service Quality Attributes at Different Levels of Operation in Higher Education Institutions: How Could the Service Quality Perceptions of Students Contribute to a Better Understanding of Improvement Directions?. Zagreb International Review of Economics and Business, S1, 1-15. https://doi.org/10.2478/zireb-2022-0021
- Tan, AH.T., Muskat, B., & Zehrer, A. (2016). A Systematic Review of Quality of Student Experience in Higher Education. International Journal of Quality and Service Sciences, 8(2), 209-228. https://doi.org/10.1108/IJQSS-08-2015-0058
- Tight, M. (2012). Researching Higher Education. McGraw-Hill Education: Maidenhead, UK.
- Tóth, Zs., & Surman, V. (2019). Listening to the voice of students, developing a service quality measuring and evaluating framework for a special course. International Journal of Quality and Service Sciences, 11(4), 455-472. https://doi.org/10.1108/IJQSS-02-2019-0025
- Wang, T., Lund, B.D., Marengo, A., Pagano, A., Mannuru, N.R., Teel, Z., & Pange, J. (2023). Exploring the Potential Impact of Artificial Intelligence (AI) on International Students in Higher Education: Generative AI, Chatbots, Analytics, and International Student Success. Applied Sciences, 13(11), 6716. https://doi.org/10.3390/app13116716
- Weichbroth, P. (2020). Usability of Mobile Applications: A Systematic Literature Study. IEEE Access, 8, 55563-55577. https://doi.org/10.1109/ACCESS.2020.2981892
- Westman, S., Kauttonen, J., Klemetti, A., Korhonen, N., Manninen, M., Mononen, A., Niittymaki, S., & Paananen, H. (2021). Artificial Intelligence for Career Guidance – Current Requirements and Prospects for the Future, IAFOR Journal of Education, 9(4), 43-62. https://doi.org/10.22492/ije.9.4.03
- Williams, R.T. (2024). The ethical implications of using generative chatbots in higher education. Frontiers in Education, 8, 1331607. https://doi.org/10.3389/feduc.2023.1331607
- Yan, T., Zhang, Y., & Hua, R. (2023). Artificial Intelligence Enterprise Service Quality Evaluation System. In 2023 Asia Conference on Advanced Robotics, Automation, and Control Engineering (ARACE), Chengdu, China, 23-28. https://doi.org/10.1109/ARACE60380.2023.00012
- Yang, S., & Evans, C. (2019). Opportunities and Challenges in Using AI Chatbots in Higher Education. Proceedings of the 2019 3rd International Conference on Education and E-Learning. https://doi.org/10.1145/3371647.3371659
- Yong, A.G., & Pearce, S. (2013). A Beginner’s Guide to Factor Analysis: Focusing on Exploratory Factor Analysis. Tutorials in Quantitative Methods for Psychology, 9(2), 79-94. https://doi.org/10.20982/TQMP.09.2.P079