Learning-Based, Automatic 2D-To-3D Image And Video Conversion

Abstract

This work is to present a new method based on the radically different approach of learning the 2D-to-3D conversion from examples. It is based on locally estimating the entire depth map of query image directly from a repository of 3D images using a nearest neighbor regression type idea. Among 2D-to-3D conversion methods, those involving human operators have been most successful but also time consuming and costly. Automatic methods that typically make use of a deterministic 3D scene model, have not yet achieved the same level of quality as they often rely on assumptions that are easily violated in practice. Despite a significant growth in the last few years, the availability of 3D content is still dwarfed by that of its 2D counterpart. To close this gap, many 2D-to-3D image and video conversion methods have been proposed. In this paper we adopt the radically different approach of learning the 3D scene structure. We develop a simplified and computationally efficient version of our recent 2D-to-3Dconversion algorithm. A repository of 3D images, either as stereo pairs or image+depth pairs, we find K pairs whose photometric content most closely matches that of a 2D query to be converted. Then, we fuse the K corresponding depth fields and align the fused depth with the 2D query.