Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos

Qiao, Tanqiu; Li, Ruochen; Li, Frederick W. B.; Kubotani, Yoshiki; Morishima, Shigeo; Shum, Hubert P. H.

Abstract

Human-Object Interaction (HOI) recognition in videos requires understanding both visual patterns and geometric relationships as they evolve over time. Visual and geometric features offer complementary strengths. Visual features capture appearance context, while geometric features provide structural patterns. Effectively fusing these multimodal features without compromising their unique characteristics remains challenging. We observe that establishing robust, entity-specific representations before modeling interactions helps preserve the strengths of each modality. Therefore, we hypothesize that a bottom-up approach is crucial for effective multimodal fusion. Following this insight, we propose the Geometric Visual Fusion Graph Neural Network (GeoVis-GNN), which uses dual-attention feature fusion combined with interdependent entity graph learning. It progressively builds from entity-specific representations toward high-level interaction understanding. To advance HOI recognition to real-world scenarios, we introduce the Concurrent Partial Interaction Dataset (MPHOI-120). It captures dynamic multi-person interactions involving concurrent actions and partial engagement. This dataset helps address challenges like complex human-object dynamics and mutual occlusions. Extensive experiments demonstrate the effectiveness of our method across various HOI scenarios. These scenarios include two-person interactions, single-person activities, bimanual manipulations, and complex concurrent partial interactions. Our method achieves state-of-the-art performance.

Citation

Qiao, T., Li, R., Li, F. W. B., Kubotani, Y., Morishima, S., & Shum, H. P. H. (2025). Geometric Visual Fusion Graph Neural Networks for Multi-Person Human-Object Interaction Recognition in Videos. Expert Systems with Applications, 290, Article 128344. https://doi.org/10.1016/j.eswa.2025.128344

• Journal Article Type: Article
• Acceptance Date: May 24, 2025
• Online Publication Date: Jun 3, 2025
• Publication Date: Sep 25, 2025
• Deposit Date: May 27, 2025
• Publicly Available Date: Jun 6, 2025
• Journal: Expert Systems with Applications
• Print ISSN: 0957-4174
• Publisher: Elsevier
• Peer Reviewed: Peer Reviewed
• Volume: 290
• Article Number: 128344
• DOI: https://doi.org/10.1016/j.eswa.2025.128344
• Public URL: https://durham-repository.worktribe.com/output/3964345