Skip to main content

Research Repository

Advanced Search

Building a three-level multimodal emotion recognition framework

Garcia-Garcia, Jose Maria; Lozano, Maria Dolores; Penichet, Victor M.R.; Law, Effie Lai-Chong

Building a three-level multimodal emotion recognition framework Thumbnail


Jose Maria Garcia-Garcia

Maria Dolores Lozano

Victor M.R. Penichet


Multimodal emotion detection has been one of the main lines of research in the field of Affective Computing (AC) in recent years. Multimodal detectors aggregate information coming from different channels or modalities to determine what emotion users are expressing with a higher degree of accuracy. However, despite the benefits offered by this kind of detectors, their presence in real implementations is still scarce for various reasons. In this paper, we propose a technology-agnostic framework, HERA, to facilitate the creation of multimodal emotion detectors, offering a tool characterized by its modularity and the interface-based programming approach adopted in its development. HERA (Heterogeneous Emotional Results Aggregator) offers an architecture to integrate different emotion detection services and aggregate their heterogeneous results to produce a final result using a common format. This proposal constitutes a step forward in the development of multimodal detectors, providing an architecture to manage different detectors and fuse the results produced by them in a sensible way. We assessed the validity of the proposal by testing the system with several developers with no previous knowledge about affective technology and emotion detection. The assessment was performed applying the Computer System Usability Questionnaire and the Twelve Cognitive Dimensions Questionnaire, used by The Visual Studio Usability group at Microsoft, obtaining positive results and important feedback for future versions of the system.


Garcia-Garcia, J. M., Lozano, M. D., Penichet, V. M., & Law, E. L. (2023). Building a three-level multimodal emotion recognition framework. Multimedia Tools and Applications, 82(1), 239-269.

Journal Article Type Article
Acceptance Date May 15, 2022
Online Publication Date Jun 6, 2022
Publication Date 2023-01
Deposit Date Jun 16, 2022
Publicly Available Date Mar 10, 2023
Journal Multimedia Tools and Applications
Print ISSN 1380-7501
Electronic ISSN 1573-7721
Publisher Springer
Peer Reviewed Peer Reviewed
Volume 82
Issue 1
Pages 239-269


Published Journal Article (1.7 Mb)

Publisher Licence URL

Copyright Statement
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

You might also like

Downloadable Citations