Building a three-level multimodal emotion recognition framework

Garcia-Garcia, Jose Maria; Lozano, Maria Dolores; Penichet, Victor M.R.; Law, Effie Lai-Chong

Abstract

Multimodal emotion detection has been one of the main lines of research in the field of Affective Computing (AC) in recent years. Multimodal detectors aggregate information coming from different channels or modalities to determine what emotion users are expressing with a higher degree of accuracy. However, despite the benefits offered by this kind of detectors, their presence in real implementations is still scarce for various reasons. In this paper, we propose a technology-agnostic framework, HERA, to facilitate the creation of multimodal emotion detectors, offering a tool characterized by its modularity and the interface-based programming approach adopted in its development. HERA (Heterogeneous Emotional Results Aggregator) offers an architecture to integrate different emotion detection services and aggregate their heterogeneous results to produce a final result using a common format. This proposal constitutes a step forward in the development of multimodal detectors, providing an architecture to manage different detectors and fuse the results produced by them in a sensible way. We assessed the validity of the proposal by testing the system with several developers with no previous knowledge about affective technology and emotion detection. The assessment was performed applying the Computer System Usability Questionnaire and the Twelve Cognitive Dimensions Questionnaire, used by The Visual Studio Usability group at Microsoft, obtaining positive results and important feedback for future versions of the system.

Citation

Garcia-Garcia, J. M., Lozano, M. D., Penichet, V. M., & Law, E. L. (2023). Building a three-level multimodal emotion recognition framework. Multimedia Tools and Applications, 82(1), 239-269. https://doi.org/10.1007/s11042-022-13254-8

• Journal Article Type: Article
• Acceptance Date: May 15, 2022
• Online Publication Date: Jun 6, 2022
• Publication Date: 2023-01
• Deposit Date: Jun 16, 2022
• Publicly Available Date: Mar 10, 2023
• Journal: Multimedia Tools and Applications
• Print ISSN: 1380-7501
• Electronic ISSN: 1573-7721
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Volume: 82
• Issue: 1
• Pages: 239-269
• DOI: https://doi.org/10.1007/s11042-022-13254-8
• Public URL: https://durham-repository.worktribe.com/output/1200782