Focalized Contrastive View-invariant Learning for Self-supervised Skeleton-based Action Recognition

Men, Qianhui; Ho, Edmond S.L.; Shum, Hubert P.H.; Leung, Howard

Abstract

Learning view-invariant representation is a key to improving feature discrimination power for skeleton-based action recognition. Existing approaches cannot effectively remove the impact of viewpoint due to the implicit view-dependent representations. In this work, we propose a self-supervised framework called Focalized Contrastive View-invariant Learning (FoCoViL), which significantly suppresses the view-specific information on the representation space where the viewpoints are coarsely aligned. By maximizing mutual information with an effective contrastive loss between multi-view sample pairs, FoCoViL associates actions with common view-invariant properties and simultaneously separates the dissimilar ones. We further propose an adaptive focalization method based on pairwise similarity to enhance contrastive learning for a clearer cluster boundary in the learned space. Different from many existing self-supervised representation learning work that rely heavily on supervised classifiers, FoCoViL performs well on both unsupervised and supervised cla ssifiers with superior recognition performance. Extensive experiments also show that the proposed contrastive-based focalization generates a more discriminative latent representation.

Citation

Men, Q., Ho, E. S., Shum, H. P., & Leung, H. (2023). Focalized Contrastive View-invariant Learning for Self-supervised Skeleton-based Action Recognition. Neurocomputing, 537, 198-209. https://doi.org/10.1016/j.neucom.2023.03.070

• Journal Article Type: Article
• Acceptance Date: Mar 28, 2023
• Online Publication Date: Mar 31, 2023
• Publication Date: Jun 7, 2023
• Deposit Date: Apr 3, 2023
• Publicly Available Date: Apr 1, 2024
• Journal: Neurocomputing
• Print ISSN: 0925-2312
• Electronic ISSN: 1872-8286
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 537
• Pages: 198-209
• DOI: https://doi.org/10.1016/j.neucom.2023.03.070
• Public URL: https://durham-repository.worktribe.com/output/1176534