Conversational Breakdown in a Customer Service Chatbot: Impact of Task Order and Criticality on User Trust and Emotion

Følstad, Asbjørn; Law, Effie L-C.; van As, Nena

Abstract

While chatbots are increasingly used for customer service, there is a knowledge gap concerning the impact of Conversational Breakdown in such chatbot interactions. In a 2 × 4 factorial design online experiment, we studied how Conversational Breakdown impacts user emotion and trust in a chatbot for customer service, given variations in task criticality and breakdown task order. Here, 257 participants were randomly assigned to complete high-or low-criticality tasks with a prototype chatbot for customer service, experiencing Conversational Breakdown for the first, second, third or none of their tasks. The task set was decided from a 63-participant pre-study. We found significant impact of Conversational Breakdown, including a marked order effect on overall trust, as well as a bounce-back effect on task-specific trust and emotion after subsequent successful task completion. We found no post-interaction effect of Task Criticality. Based on our findings, we discuss theoretical and practical implications and suggest future research.

Citation

Følstad, A., Law, E. L.-C., & van As, N. (2024). Conversational Breakdown in a Customer Service Chatbot: Impact of Task Order and Criticality on User Trust and Emotion. ACM Transactions on Computer-Human Interaction, 31(5), 1-52. https://doi.org/10.1145/3690383

• Journal Article Type: Article
• Acceptance Date: Jul 31, 2024
• Online Publication Date: Sep 3, 2024
• Publication Date: 2024-10
• Deposit Date: Sep 26, 2024
• Publicly Available Date: Sep 26, 2024
• Journal: ACM Transactions on Computer-Human Interaction
• Print ISSN: 1073-0516
• Electronic ISSN: 1557-7325
• Publisher: Association for Computing Machinery (ACM)
• Peer Reviewed: Peer Reviewed
• Volume: 31
• Issue: 5
• Article Number: 66
• Pages: 1-52
• DOI: https://doi.org/10.1145/3690383
• Keywords: CCS Concepts:; Human-centred computing; Human-computer interaction (HCI); Interaction paradigms; Natural language interfaces; Additional Key Words and Phrases: Conversational agent, Conversational Breakdown, Task Criticality, Trust, AQ3
• Public URL: https://durham-repository.worktribe.com/output/2876950