Kanglei Zhou
STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising
Zhou, Kanglei; Cheng, Zhiyuan; Shum, Hubert P.H.; Li, Frederick W.B.; Liang, Xiaohui
Authors
Zhiyuan Cheng
Professor Hubert Shum hubert.shum@durham.ac.uk
Professor
Dr Frederick Li frederick.li@durham.ac.uk
Associate Professor
Xiaohui Liang
Abstract
Hand object interaction in mixed reality (MR) relies on the accurate tracking and estimation of human hands, which provide users with a sense of immersion. However, raw captured hand motion data always contains errors such as joints occlusion, dislocation, high-frequency noise, and involuntary jitter. Denoising and obtaining the hand motion data consistent with the user’s intention are of the utmost importance to enhance the interactive experience in MR. To this end, we propose an end-to-end method for hand motion denoising using the spatial-temporal graph auto-encoder (STGAE). The spatial and temporal patterns are recognized simultaneously by constructing the consecutive hand joint sequence as a spatial-temporal graph. Considering the complexity of the articulated hand structure, a simple yet effective partition strategy is proposed to model the physic-connected and symmetry-connected relationships. Graph convolution is applied to extract structural constraints of the hand, and a self-attention mechanism is to adjust the graph topology dynamically. Combining graph convolution and temporal convolution, a fundamental graph encoder or decoder block is proposed. We finally establish the hourglass residual auto-encoder to learn a manifold projection operation and a corresponding inverse projection through stacking these blocks. In this work, the proposed framework has been successfully used in hand motion data denoising with preserving structural constraints between joints. Extensive quantitative and qualitative experiments show that the proposed method has achieved better performance than the state-of-the-art approaches.
Citation
Zhou, K., Cheng, Z., Shum, H. P., Li, F. W., & Liang, X. (2021, October). STGAE: Spatial-Temporal Graph Auto-Encoder for Hand Motion Denoising. Presented at 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), Bari, Italy
Presentation Conference Type | Conference Paper (published) |
---|---|
Conference Name | 2021 IEEE International Symposium on Mixed and Augmented Reality (ISMAR) |
Start Date | Oct 4, 2021 |
End Date | Oct 8, 2021 |
Acceptance Date | Aug 10, 2021 |
Online Publication Date | Nov 1, 2021 |
Publication Date | 2021 |
Deposit Date | Aug 19, 2021 |
Publicly Available Date | Oct 9, 2021 |
Publisher | Institute of Electrical and Electronics Engineers |
DOI | https://doi.org/10.1109/ismar52148.2021.00018 |
Public URL | https://durham-repository.worktribe.com/output/1140509 |
Files
Accepted Conference Proceeding
(1.2 Mb)
PDF
Copyright Statement
© 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
You might also like
RAPiD-Seg: Range-Aware Pointwise Distance Distribution Networks for 3D LiDAR Segmentation
(2024)
Presentation / Conference Contribution
Two-Person Interaction Augmentation with Skeleton Priors
(2024)
Presentation / Conference Contribution
One-Index Vector Quantization Based Adversarial Attack on Image Classification
(2024)
Journal Article
Downloadable Citations
About Durham Research Online (DRO)
Administrator e-mail: dro.admin@durham.ac.uk
This application uses the following open-source libraries:
SheetJS Community Edition
Apache License Version 2.0 (http://www.apache.org/licenses/)
PDF.js
Apache License Version 2.0 (http://www.apache.org/licenses/)
Font Awesome
SIL OFL 1.1 (http://scripts.sil.org/OFL)
MIT License (http://opensource.org/licenses/mit-license.html)
CC BY 3.0 ( http://creativecommons.org/licenses/by/3.0/)
Powered by Worktribe © 2024
Advanced Search