Skip to main content

Research Repository

Advanced Search

STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

Zhou, Kanglei; Cheng, Zhiyuan; Shum, Hubert P.H.; Li, Frederick W.B.; Liang, Xiaohui

STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising Thumbnail


Kanglei Zhou

Zhiyuan Cheng

Xiaohui Liang


Hand object interaction in mixed reality (MR) relies on the accurate tracking and estimation of human hands, which provide users with a sense of immersion. However, raw captured hand motion data always contains errors such as joints occlusion, dislocation, high-frequency noise, and involuntary jitter. Denoising and obtaining the hand motion data consistent with the user’s intention are of the utmost importance to enhance the interactive experience in MR. To this end, we propose an end-to-end method for hand motion denoising using the spatial-temporal graph auto-encoder (STGAE). The spatial and temporal patterns are recognized simultaneously by constructing the consecutive hand joint sequence as a spatial-temporal graph. Considering the complexity of the articulated hand structure, a simple yet effective partition strategy is proposed to model the physic-connected and symmetry-connected relationships. Graph convolution is applied to extract structural constraints of the hand, and a self-attention mechanism is to adjust the graph topology dynamically. Combining graph convolution and temporal convolution, a fundamental graph encoder or decoder block is proposed. We finally establish the hourglass residual auto-encoder to learn a manifold projection operation and a corresponding inverse projection through stacking these blocks. In this work, the proposed framework has been successfully used in hand motion data denoising with preserving structural constraints between joints. Extensive quantitative and qualitative experiments show that the proposed method has achieved better performance than the state-of-the-art approaches.


Zhou, K., Cheng, Z., Shum, H. P., Li, F. W., & Liang, X. (2021). STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising. .

Conference Name 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
Conference Location Bari, Italy
Start Date Oct 4, 2021
End Date Oct 8, 2021
Acceptance Date Aug 10, 2021
Online Publication Date Nov 1, 2021
Publication Date 2021
Deposit Date Aug 19, 2021
Publicly Available Date Oct 9, 2021
Publisher Institute of Electrical and Electronics Engineers


Accepted Conference Proceeding (1.2 Mb)

Copyright Statement
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.

You might also like

Downloadable Citations