Is Unimodal Bias Always Bad for Visual Question Answering? A Medical Domain Study with Dynamic Attention

Sun, Zhongtian; Harit, Anoushka; Cristea, Alexandra I.; Yu, Jialin; Al Moubayed, Noura; Shi, Lei

Abstract

Medical visual question answering (Med-VQA) is to answer medical questions based on clinical images provided. This field is still in its infancy due to the complexity of the trio formed of questions, multimodal features and expert knowledge. In this paper, we tackle, a ’myth’ in the Natural Language Processing area - that unimodal bias is always considered undesirable in learning models. Additionally, we study the effect of integrating a novel dynamic attention mechanism into such models, inspired by a recent graph deep learning study.Unlike traditional attention, dynamic attention scores are conditioned on different query words in a question and thus enhance the representation learning ability of texts. We propose that some questions are answered more accurately with a reinforcement of question embedding after fusing multimodal features. Extensive experiments have been implemented on the VQA-RAD datasets and demonstrate that our proposed model, reinforCe unimOdal dynamiC Attention (COCA), outperforms the state-of-the-art methods overall and performs competitively at open-ended question answering.

Citation

Sun, Z., Harit, A., Cristea, A. I., Yu, J., Al Moubayed, N., & Shi, L. (2022). Is Unimodal Bias Always Bad for Visual Question Answering? A Medical Domain Study with Dynamic Attention. . https://doi.org/10.1109/bigdata55660.2022.10020791

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: IEEE Big Data
• Start Date: Dec 17, 2022
• End Date: Dec 20, 2022
• Acceptance Date: Oct 18, 2022
• Online Publication Date: Jan 26, 2023
• Publication Date: 2022
• Deposit Date: Oct 20, 2022
• Publicly Available Date: Dec 6, 2022
• DOI: https://doi.org/10.1109/bigdata55660.2022.10020791
• Public URL: https://durham-repository.worktribe.com/output/1135472