Skip to main content

Research Repository

Advanced Search

STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising

Zhou, Kanglei; Cheng, Zhiyuan; Shum, Hubert P.H.; Li, Frederick W.B.; Liang, Xiaohui

STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising Thumbnail


Authors

Kanglei Zhou

Zhiyuan Cheng

Xiaohui Liang



Abstract

Hand object interaction in mixed reality (MR) relies on the accurate tracking and estimation of human hands, which provide users with a sense of immersion. However, raw captured hand motion data always contains errors such as joints occlusion, dislocation, high-frequency noise, and involuntary jitter. Denoising and obtaining the hand motion data consistent with the user’s intention are of the utmost importance to enhance the interactive experience in MR. To this end, we propose an end-to-end method for hand motion denoising using the spatial-temporal graph auto-encoder (STGAE). The spatial and temporal patterns are recognized simultaneously by constructing the consecutive hand joint sequence as a spatial-temporal graph. Considering the complexity of the articulated hand structure, a simple yet effective partition strategy is proposed to model the physic-connected and symmetry-connected relationships. Graph convolution is applied to extract structural constraints of the hand, and a self-attention mechanism is to adjust the graph topology dynamically. Combining graph convolution and temporal convolution, a fundamental graph encoder or decoder block is proposed. We finally establish the hourglass residual auto-encoder to learn a manifold projection operation and a corresponding inverse projection through stacking these blocks. In this work, the proposed framework has been successfully used in hand motion data denoising with preserving structural constraints between joints. Extensive quantitative and qualitative experiments show that the proposed method has achieved better performance than the state-of-the-art approaches.

Citation

Zhou, K., Cheng, Z., Shum, H. P., Li, F. W., & Liang, X. (2021, October). STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising. Presented at 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Bari, Italy

Presentation Conference Type Conference Paper (published)
Conference Name 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
Start Date Oct 4, 2021
End Date Oct 8, 2021
Acceptance Date Aug 10, 2021
Online Publication Date Nov 1, 2021
Publication Date 2021
Deposit Date Aug 19, 2021
Publicly Available Date Oct 9, 2021
Publisher Institute of Electrical and Electronics Engineers
DOI https://doi.org/10.1109/ismar52148.2021.00018
Public URL https://durham-repository.worktribe.com/output/1140509

Files

Accepted Conference Proceeding (1.2 Mb)
PDF

Copyright Statement
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.






You might also like



Downloadable Citations