Enhancing Perception and Immersion in Pre-Captured Environments through Learning-Based Eye Height Adaptation

Feng, Qi; Shum, Hubert P.H.; Morishima, Shigeo

Abstract

Pre-captured immersive environments using omnidirectional cameras provide a wide range of virtual reality applications. Previous research has shown that manipulating the eye height in egocentric virtual environments can significantly affect distance perception and immersion. However, the influence of eye height in pre-captured real environments has received less attention due to the difficulty of altering the perspective after finishing the capture process. To explore this influence, we first propose a pilot study that captures real environments with multiple eye heights and asks participants to judge the egocentric distances and immersion. If a significant influence is confirmed, an effective image-based approach to adapt pre-captured real-world environments to the user’s eye height would be desirable. Motivated by the study, we propose a learning-based approach for synthesizing novel views for omnidirectional images with altered eye heights. This approach employs a multitask architecture that learns depth and semantic segmentation in two formats, and generates high-quality depth and semantic segmentation to facilitate the inpainting stage. With the improved omnidirectional-aware layered depth image, our approach synthesizes natural and realistic visuals for eye height adaptation. Quantitative and qualitative evaluation shows favorable results against state-of-the-art methods, and an extensive user study verifies improved perception and immersion for pre-captured real-world environments.

Citation

Feng, Q., Shum, H. P., & Morishima, S. (2023). Enhancing Perception and Immersion in Pre-Captured Environments through Learning-Based Eye Height Adaptation. In 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR). https://doi.org/10.1109/ISMAR59233.2023.00055

• Presentation Conference Type: Conference Paper (Published)
• Conference Name: ISMAR 23: International Symposium on Mixed and Augmented Reality
• Start Date: Oct 16, 2023
• End Date: Oct 20, 2023
• Acceptance Date: Aug 10, 2023
• Online Publication Date: Dec 4, 2023
• Publication Date: Dec 4, 2023
• Deposit Date: Aug 16, 2023
• Publicly Available Date: Dec 4, 2023
• Publisher: Institute of Electrical and Electronics Engineers
• Series ISSN: 1554-7868
• Book Title: 2023 IEEE International Symposium on Mixed and Augmented Reality (ISMAR)
• ISBN: 9798350328394
• DOI: https://doi.org/10.1109/ISMAR59233.2023.00055
• Public URL: https://durham-repository.worktribe.com/output/1718711