Abstract

We present 3DLite, a novel approach to reconstruct 3D environments using consumer RGB-D sensors, making a step towards directly utilizing captured 3D content in graphics applications, such as video games, VR, or AR. Rather than reconstructing an accurate one-to-one representation of the real world, our method computes a lightweight, low-polygonal geometric abstraction of the scanned geometry. We argue that for many graphics applications it is much more important to obtain high-quality surface textures rather than highly-detailed geometry. To this end, we compensate for motion blur, auto-exposure artifacts, and micro-misalignments in camera poses by warping and stitching image fragments from low-quality RGB input data to achieve high-resolution, sharp surface textures. In addition to the observed regions of a scene, we extrapolate the scene geometry, as well as the mapped surface textures, to obtain a complete 3D model of the environment. We show that a simple planar abstraction of the scene geometry is ideally suited for this completion task, enabling 3DLite to produce complete, lightweight, and visually compelling 3D scene models. We believe that these CAD-like reconstructions are an important step towards leveraging RGB-D scanning in actual content creation pipelines.

Paper HQ | Paper LQ | Supplemental | Dataset | BibTeX citation | Source Code (Coming Soon)

We provide 3DLite models for BundleFusion Scenes office0, office1, office3, as well as ScanNet sequences 0567_01, 0451_05, 0294_02, 0271_01, 0220_02. We also provide 3DLite models and rgb-d data for two additional sequences captured by a Structure.io depth sensor coupled with an iPad color camera. Please refer to the respective publication when using this data.

Format

3DLite models are provided as zipped obj files.
Rgb-d sequences each contain:

• Color frames (frame-XXXXXX.color.jpg): RGB, 24-bit, JPG


• Depth frames (frame-XXXXXX.depth.png): depth (mm), 16-bit, PNG (invalid depth is set to 0)


• Camera poses (frame-XXXXXX.pose.txt): camera-to-world (invalid transforms -INF)


• Camera calibration (info.txt): color and depth camera intrinsics and extrinsics.

We also have the above data in our custom .sens format, please see the c++ reader for how to load the file.

License

The data has been released under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 License.

BundleFusion office3		office3-3dlite.zip
BundleFusion office1		office1-3dlite.zip
BundleFusion office0		office0-3dlite.zip
ScanNet scene0567_01		scene0567_01-3dlite.zip
ScanNet scene0451_05		scene0451_05-3dlite.zip
ScanNet scene0294_02		scene0294_02-3dlite.zip
ScanNet scene0271_01		scene0271_01-3dlite.zip
ScanNet scene0220_02		scene0220_02-3dlite.zip
apt	640x480 color 640x480 depth	apt-3dlite.zip apt-initial.ply apt-rgbd.zip (745MB) apt.sens (1.3GB)
offices	640x480 color 640x480 depth	offices-3dlite.zip offices-initial.ply offices-rgbd.zip (3.6GB) offices.sens (5.1GB)