Stable Hand Pose Estimation under Tremor via Graph Neural Network

Leng, Zhiying; Chen, Jiaying; Shum, Hubert P.H.; Li, Frederick W.B.; Liang, Xiaohui

Abstract

Hand pose estimation, which predicts the spatial location of hand joints, is a fundamental task in VR/AR applications. Although existing methods can recover hand pose competently, the tremor issue occurring in hand motion has not been completely solved. Tremor is an involuntary motion accompanied by a desired gesture or hand motion, leading to hand pose that deviates from user's intentions. Considering the characteristic of tremor motion, we present a novel Graph Neural Network for stable 3D hand pose estimation. The input is depth images. The constraint adjacency matrix is devised in Graph Neural Network for dynamically adjusting the topology of a hand graph during message passing and aggregation. Firstly, since there are rich potential constraints among hand joints, we utilize the constraint adjacency matrix to mine the suitable topology, modeling spatial-temporal constraints of joints and outputting the precise tremor hand pose as the pre-estimation result. Then, for obtaining a stable hand pose, we provide a tremor compensation module based on the constraint adjacency matrix, which exploits the constraint between control points and tremor hand pose. Concretely, the control points represented the voluntary motion are employed as constraints to edit the tremor hand pose. Our extensive quantitative and qualitative experiments show that the proposed method has achieved decent performance for 3D tremor hand pose estimation.

Citation

Leng, Z., Chen, J., Shum, H. P., Li, F. W., & Liang, X. (2021, March). Stable Hand Pose Estimation under Tremor via Graph Neural Network. Presented at 2021 IEEE Virtual Reality and 3D User Interfaces (VR), Lisboa

• Presentation Conference Type: Conference Paper (published)
• Conference Name: 2021 IEEE Virtual Reality and 3D User Interfaces (VR)
• Start Date: Mar 27, 2021
• End Date: Apr 1, 2021
• Acceptance Date: Jan 14, 2021
• Online Publication Date: May 10, 2021
• Publication Date: 2021
• Deposit Date: Mar 22, 2021
• Publicly Available Date: Jul 31, 2023
• Pages: 226-234
• Series ISSN: 2642-5246
• Book Title: 2021 IEEE Virtual Reality and 3D User Interfaces (VR)
• DOI: https://doi.org/10.1109/vr50410.2021.00044
• Public URL: https://durham-repository.worktribe.com/output/1139660
• Additional Information: Date of Conference: 27 March 2021 - 01 April 2021