Dr Parminder Kaur parminder.kaur@durham.ac.uk
Post Doctoral Research Associate
Annotate and retrieve in vivo images using hybrid self-organizing map
Kaur, Parminder; Malhi, Avleen; Pannu, Husanbir
Authors
Avleen Malhi
Husanbir Pannu
Abstract
Multimodal retrieval has gained much attention lately due to its effectiveness over uni-modal retrieval. For instance, visual features often under-constrain the description of an image in content-based retrieval; however, another modality, such as collateral text, can be introduced to abridge the semantic gap and make the retrieval process more efficient. This article proposes the application of cross-modal fusion and retrieval on real in vivo gastrointestinal images and linguistic cues, as the visual features alone are insufficient for image description and to assist gastroenterologists. So, a cross-modal information retrieval approach has been proposed to retrieve related images given text and vice versa while handling the heterogeneity gap issue among the modalities. The technique comprises two stages: (1) individual modality feature learning; and (2) fusion of two trained networks. In the first stage, two self-organizing maps (SOMs) are trained separately using images and texts, which are clustered in the respective SOMs based on their similarity. In the second (fusion) stage, the trained SOMs are integrated using an associative network to enable cross-modal retrieval. The underlying learning techniques of the associative network include Hebbian learning and Oja learning (Improved Hebbian learning). The introduced framework can annotate images with keywords and illustrate keywords with images, and it can also be extended to incorporate more diverse modalities. Extensive experimentation has been performed on real gastrointestinal images obtained from a known gastroenterologist that have collateral keywords with each image. The obtained results proved the efficacy of the algorithm and its significance in aiding gastroenterologists in quick and pertinent decision making.
Citation
Kaur, P., Malhi, A., & Pannu, H. (2024). Annotate and retrieve in vivo images using hybrid self-organizing map. Visual Computer, 40(8), 5619-5638. https://doi.org/10.1007/s00371-023-03126-z
Journal Article Type | Article |
---|---|
Acceptance Date | Sep 23, 2023 |
Online Publication Date | Oct 31, 2023 |
Publication Date | Aug 1, 2024 |
Deposit Date | Nov 7, 2023 |
Publicly Available Date | Nov 7, 2023 |
Journal | The Visual Computer |
Print ISSN | 0178-2789 |
Electronic ISSN | 1432-2315 |
Publisher | Springer |
Peer Reviewed | Peer Reviewed |
Volume | 40 |
Issue | 8 |
Pages | 5619-5638 |
DOI | https://doi.org/10.1007/s00371-023-03126-z |
Keywords | Self-organizing map, Cross-modal retrieval, Hebbian learning, Oja rule, Gastrointestinal endoscopy |
Public URL | https://durham-repository.worktribe.com/output/1899224 |
Files
Published Journal Article (Advance Online Version)
(3.7 Mb)
PDF
Licence
http://creativecommons.org/licenses/by/4.0/
Publisher Licence URL
http://creativecommons.org/licenses/by/4.0/
Copyright Statement
This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
You might also like
Segmentation and Identification of Mediterranean Plant Species
(2023)
Presentation / Conference Contribution
Downloadable Citations
About Durham Research Online (DRO)
Administrator e-mail: dro.admin@durham.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search