Annotate and retrieve in vivo images using hybrid self-organizing map

Kaur, Parminder; Malhi, Avleen; Pannu, Husanbir

Abstract

Multimodal retrieval has gained much attention lately due to its effectiveness over uni-modal retrieval. For instance, visual features often under-constrain the description of an image in content-based retrieval; however, another modality, such as collateral text, can be introduced to abridge the semantic gap and make the retrieval process more efficient. This article proposes the application of cross-modal fusion and retrieval on real in vivo gastrointestinal images and linguistic cues, as the visual features alone are insufficient for image description and to assist gastroenterologists. So, a cross-modal information retrieval approach has been proposed to retrieve related images given text and vice versa while handling the heterogeneity gap issue among the modalities. The technique comprises two stages: (1) individual modality feature learning; and (2) fusion of two trained networks. In the first stage, two self-organizing maps (SOMs) are trained separately using images and texts, which are clustered in the respective SOMs based on their similarity. In the second (fusion) stage, the trained SOMs are integrated using an associative network to enable cross-modal retrieval. The underlying learning techniques of the associative network include Hebbian learning and Oja learning (Improved Hebbian learning). The introduced framework can annotate images with keywords and illustrate keywords with images, and it can also be extended to incorporate more diverse modalities. Extensive experimentation has been performed on real gastrointestinal images obtained from a known gastroenterologist that have collateral keywords with each image. The obtained results proved the efficacy of the algorithm and its significance in aiding gastroenterologists in quick and pertinent decision making.

Citation

Kaur, P., Malhi, A., & Pannu, H. (2024). Annotate and retrieve in vivo images using hybrid self-organizing map. Visual Computer, 40(8), 5619-5638. https://doi.org/10.1007/s00371-023-03126-z

• Journal Article Type: Article
• Acceptance Date: Sep 23, 2023
• Online Publication Date: Oct 31, 2023
• Publication Date: Aug 1, 2024
• Deposit Date: Nov 7, 2023
• Publicly Available Date: Nov 7, 2023
• Journal: The Visual Computer
• Print ISSN: 0178-2789
• Electronic ISSN: 1432-2315
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Volume: 40
• Issue: 8
• Pages: 5619-5638
• DOI: https://doi.org/10.1007/s00371-023-03126-z
• Keywords: Self-organizing map, Cross-modal retrieval, Hebbian learning, Oja rule, Gastrointestinal endoscopy
• Public URL: https://durham-repository.worktribe.com/output/1899224