3D Sketching in Virtual Reality for immersive model search

Giunchi, Daniele; James, Stuart; Steed, Anthony

Abstract

We describe a novel method for searching 3D model collections using free-form sketches within a virtual environment as queries. As opposed to traditional sketch retrieval, our queries are drawn directly onto an example model. Using immersive virtual reality the user can express their query through a sketch that demonstrates the desired structure, color and texture. Unlike previous sketch-based retrieval methods, users remain immersed within the environment without relying on textual queries or 2D projections which can disconnect the user from the environment. We perform a test using queries over several descriptors, evaluating the precision in order to select the most accurate one. We show how a convolutional neural network (CNN) can create multi-view representations of colored 3D sketches. Using such a descriptor representation, our system is able to rapidly retrieve models and in this way, we provide the user with an interactive method of navigating large object datasets. Through a user study we demonstrate that by using our VR 3D model retrieval system, users can perform search more quickly and intuitively than with a naive linear browsing method. Using our system users can rapidly populate a virtual environment with specific models from a very large database, and thus the technique has the potential to be broadly applicable in immersive editing systems.

Citation

Giunchi, D., James, S., & Steed, A. (2018). 3D Sketching in Virtual Reality for immersive model search. In Expressive '18: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering (1-12). https://doi.org/10.1145/3229147.3229166

• Conference Name: Expressive '18: Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering
• Conference Location: Victoria, BC, Canada
• Start Date: Aug 17, 2028
• End Date: Aug 19, 2018
• Online Publication Date: Aug 17, 2018
• Publication Date: 2018-08
• Deposit Date: Dec 13, 2023
• Publisher: Association for Computing Machinery (ACM)
• Pages: 1-12
• Book Title: Expressive '18: Proceedings of the Joint Symposium on Computational Aesthetics and Sketch-Based Interfaces and Modeling and Non-Photorealistic Animation and Rendering
• ISBN: 9781450358927
• DOI: https://doi.org/10.1145/3229147.3229166
• Public URL: https://durham-repository.worktribe.com/output/2024589