Value-sensitive Evaluation of Hybrid Human-AI Chatbots in Customer Services

Abstract

The state-of-the-art shows the potential of chatbots and other Machine Learning (ML) models to perform many tasks of high quality. Especially chatbots are already used by many companies to assist their customer service. However, chatbots will likely never be able to perform all tasks perfectly. Therefore, it is still the question whether such a chatbot is valuable for a business. Current research fails to describe how chatbots should be evaluated to compute the value of a chatbot for a business. In this research, we design an evaluation framework capturing the value of a chatbot in customer service. This framework consists of several key dimensions which should be computed in order to determine the value of the chatbot. To show that this evaluation framework captures the value of a chatbot, we perform a case study on water utility companies in The Netherlands. This case study showed the designed evaluation framework does capture the value of a chatbot in customer service.