Skip to main content

Research Repository

Advanced Search

Annotate and retrieve in vivo images using hybrid self-organizing map

Kaur, Parminder; Malhi, Avleen; Pannu, Husanbir

Annotate and retrieve in vivo images using hybrid self-organizing map Thumbnail


Avleen Malhi

Husanbir Pannu


Multimodal retrieval has gained much attention lately due to its effectiveness over uni-modal retrieval. For instance, visual features often under-constrain the description of an image in content-based retrieval; however, another modality, such as collateral text, can be introduced to abridge the semantic gap and make the retrieval process more efficient. This article proposes the application of cross-modal fusion and retrieval on real in vivo gastrointestinal images and linguistic cues, as the visual features alone are insufficient for image description and to assist gastroenterologists. So, a cross-modal information retrieval approach has been proposed to retrieve related images given text and vice versa while handling the heterogeneity gap issue among the modalities. The technique comprises two stages: (1) individual modality feature learning; and (2) fusion of two trained networks. In the first stage, two self-organizing maps (SOMs) are trained separately using images and texts, which are clustered in the respective SOMs based on their similarity. In the second (fusion) stage, the trained SOMs are integrated using an associative network to enable cross-modal retrieval. The underlying learning techniques of the associative network include Hebbian learning and Oja learning (Improved Hebbian learning). The introduced framework can annotate images with keywords and illustrate keywords with images, and it can also be extended to incorporate more diverse modalities. Extensive experimentation has been performed on real gastrointestinal images obtained from a known gastroenterologist that have collateral keywords with each image. The obtained results proved the efficacy of the algorithm and its significance in aiding gastroenterologists in quick and pertinent decision making.


Kaur, P., Malhi, A., & Pannu, H. (2023). Annotate and retrieve in vivo images using hybrid self-organizing map. Visual Computer,

Journal Article Type Article
Acceptance Date Sep 23, 2023
Online Publication Date Oct 31, 2023
Publication Date 2023
Deposit Date Nov 7, 2023
Publicly Available Date Nov 7, 2023
Journal The Visual Computer
Print ISSN 0178-2789
Electronic ISSN 1432-2315
Publisher Springer
Peer Reviewed Peer Reviewed
Keywords Computer Graphics and Computer-Aided Design; Computer Vision and Pattern Recognition; Software
Public URL


Published Journal Article (Advance Online Version) (3.7 Mb)


Publisher Licence URL

Copyright Statement
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit

You might also like

Downloadable Citations