3D car shape reconstruction from a contour sketch using GAN and lazy learning

Nozawa, Naoki; Shum, Hubert P.H.; Feng, Qi; Ho, Edmond S.L.; Morishima, Shigeo

Abstract

3D car models are heavily used in computer games, visual effects, and even automotive designs. As a result, producing such models with minimal labour costs is increasingly more important. To tackle the challenge, we propose a novel system to reconstruct a 3D car using a single sketch image. The system learns from a synthetic database of 3D car models and their corresponding 2D contour sketches and segmentation masks, allowing effective training with minimal data collection cost. The core of the system is a machine learning pipeline that combines the use of a generative adversarial network (GAN) and lazy learning. GAN, being a deep learning method, is capable of modelling complicated data distributions, enabling the effective modelling of a large variety of cars. Its major weakness is that as a global method, modelling the fine details in the local region is challenging. Lazy learning works well to preserve local features by generating a local subspace with relevant data samples. We demonstrate that the combined use of GAN and lazy learning produces is able to produce high-quality results, in which different types of cars with complicated local features can be generated effectively with a single sketch. Our method outperforms existing ones using other machine learning structures such as the variational autoencoder.

Citation

Nozawa, N., Shum, H. P., Feng, Q., Ho, E. S., & Morishima, S. (2022). 3D car shape reconstruction from a contour sketch using GAN and lazy learning. Visual Computer, 38(4), 1317-1330. https://doi.org/10.1007/s00371-020-02024-y

• Journal Article Type: Article
• Acceptance Date: Sep 5, 2020
• Online Publication Date: Apr 16, 2021
• Publication Date: 2022-04
• Deposit Date: Aug 11, 2021
• Publicly Available Date: Aug 11, 2021
• Journal: The Visual Computer
• Print ISSN: 0178-2789
• Electronic ISSN: 1432-2315
• Publisher: Springer
• Peer Reviewed: Peer Reviewed
• Volume: 38
• Issue: 4
• Pages: 1317-1330
• DOI: https://doi.org/10.1007/s00371-020-02024-y
• Public URL: https://durham-repository.worktribe.com/output/1242994