Matterport3D: Learning from RGB-D Data in Indoor Environments

Angel X. Chang, Angela Dai, Thomas Funkhouser, Maciej Halber, Matthias Nießner, Manolis Savva, Shuran Song, Andy Zeng, Yinda Zhang

3DV 2017

In this paper, we introduce Matterport3D, a large-scale RGB-D dataset containing 10,800 panoramic views from 194,400 RGB-D images of 90 building-scale scenes.

Multiframe Scene Flow with Piecewise Rigid Motion

Vladislav Golyanik, Kihwan Kim, Robert Maier, Matthias Nießner, Didier Stricker, Jan Kautz

3DV 2017

We introduce a novel multi-frame scene flow approach that jointly optimizes the consistency of the patch appearances and their local rigid motions from RGB-D image sequences.

3DLite: Towards Commodity 3D Scanning for Content Creation

Jingwei Huang, Angela Dai, Leonidas Guibas, Matthias Nießner

ACM Transactions on Graphics 2017 (TOG)

We present 3DLite, a novel approach to reconstruct 3D environments using consumer RGB-D sensors, making a step towards directly utilizing captured 3D content in graphics applications, such as video games, VR, or AR. Rather than reconstructing an accurate one-to-one representation of the real world, our method computes a lightweight, low-polygonal geometric abstraction of the scanned geometry.

Autonomous Reconstruction of Unknown Indoor Scenes Guided by Time-varying Tensor Fields

Kai Xu, Lintao Zheng, Zihao Yan, Guohang Yan, Eugene Zhang, Matthias Nießner, Oliver Deussen, Daniel Cohen-Or, Hui Huang

ACM Transactions on Graphics 2017 (TOG)

We present a navigation-by-reconstruction approach to address this question where moving paths of the robot are planned to account for both global efficiency for fast exploration and local smoothness to obtain high-quality scans.

Opt: A Domain Specific Language for Non-linear Least Squares Optimization in Graphics and Imaging

Zachary DeVito, Michael Mara, Michael Zollhöfer, Gilbert Bernstein, Jonathan Ragan-Kelley, Christian Theobalt, Pat Hanrahan, Matthew Fisher, Matthias Nießner

ACM Transactions on Graphics 2017 (TOG)

We propose a new language, Opt, in which a user simply writes energy functions over image- or graph-structured unknowns, and a compiler automatically generates state-of-the-art GPU optimization kernels.

BundleFusion: Real-time Globally Consistent 3D Reconstruction using On-the-fly Surface Re-integration

Angela Dai, Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Christian Theobalt

ACM Transactions on Graphics 2017 (TOG)

We introduce a novel, real-time, end-to-end 3D reconstruction framework, with a robust pose optimization strategy based on sparse feature matches and dense geometric and photometric alignment. One main contribution is the ability to update the reconstructed model on-the-fly as new (global) pose optimization results become available.

Intrinsic3D: High-Quality 3D Reconstruction by Joint Appearance and Geometry Optimization with Spatially-Varying Lighting

Robert Maier, Kihwan Kim, Daniel Cremers, Jan Kautz, Matthias Nießner

ICCV 2017

We introduce a novel method to obtain high-quality 3D reconstructions from consumer RGB-D sensors. Our core idea is to simultaneously optimize for geometry encoded in a signed distance field (SDF), textures from automaticallyselected keyframes, and their camera poses along with material and scene lighting.

A Lightweight Approach for On-the-Fly Reflectance Estimation

Kihwan Kim, Jinwei Gu, Stephen Tyree, Pavlo Molchanov, Matthias Nießner, Jan Kautz

ICCV 2017 (Oral)

We propose a lightweight, learning-based approach for surface reflectance estimation directly from 8-bit RGB images in real-time, which can be easily plugged into any 3D scanning-and-fusion system with a commodity RGBD sensor.

ScanNet: Richly-annotated 3D Reconstructions of Indoor Scenes

Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, Matthias Nießner

CVPR 2017 (Spotlight)

We introduce ScanNet, an RGB-D video dataset containing 2.5M views in 1513 scenes annotated with 3D camera poses, surface reconstructions, and semantic segmentations.

Shape Completion using 3D-Encoder-Predictor CNNs and Shape Synthesis

Angela Dai, Charles Ruizhongtai Qi, Matthias Nießner

CVPR 2017 (Spotlight)

We introduce a data-driven approach to complete partial 3D shapes through a combination of volumetric deep neural networks and 3D shape synthesis. From a partially-scanned input shape, our method first infers a low-resolution - but complete - output.

3DMatch: Learning the Matching of Local 3D Geometry in Range Scans

Andy Zeng, Shuran Song, Matthias Nießner, Matthew Fisher, Jianxiong Xiao

CVPR 2017 (Oral)

In this paper, we introduce 3DMatch, a data-driven local feature learner that jointly learns a geometric feature representation and an associated metric function from a large collection of real-world scanning data.

Learning to Navigate the Energy Landscape

Julien Valentin, Angela Dai, Matthias Nießner, Pushmeet Kohli, Philip H. S. Torr, Shahram Izadi, Cem Keskin

3DV 2016

In this paper, we present a novel, general, and efficient architecture for addressing computer vision problems that are approached from an `Analysis by Synthesis' standpoint.

VolumeDeform: Real-time Volumetric Non-rigid Reconstruction

Matthias Innmann, Michael Zollhöfer, Matthias Nießner, Christian Theobalt, Marc Stamminger

ECCV 2016

We present a novel approach for the reconstruction of dynamic geometric shapes using a single hand-held consumer-grade RGB-D sensor at real-time rates. Our method does not require a pre-defined shape template to start with and builds up the scene model from scratch during the scanning process.

Efficient GPU Rendering of Subdivision Surfaces using Adaptive Quadtrees

Wade Brainerd, Tim Foley, Manuel Kraemer, Henry Moreton, Matthias Nießner

ACM Transactions on Graphics 2016 (TOG)

We present a novel method for real-time rendering of subdivision surfaces whose goal is to make subdivision faces as easy to render as triangles, points, or lines. Our approach uses the GPU tessellation hardware and processes each face of a base mesh independently and in a streaming fashion, thus allowing an entire model to be rendered in a single pass.

PiGraphs: Learning Interaction Snapshots from Observations

Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner

ACM Transactions on Graphics 2016 (TOG)

We learn a probabilistic model connecting human poses and arrangements of objects from observations of interactions collected with commodity RGB-D sensors. This model is encoded as a set of Prototypical Interaction Graphs (PiGraphs): a human-centric representation capturing physical contact and attention linkages between geometry and the human body.

ProxImaL: Efficient Image Optimization using Proximal Algorithms

Felix Heide, Steven Diamond, Matthias Nießner, Jonathan Ragan-Kelley, Wolfgang Heidrich, Gordon Wetzstein

ACM Transactions on Graphics 2016 (TOG)

ProxImaL is a domain-specific language and compiler for image optimization problems that makes it easy to experiment with different problem formulations and algorithm choices. The compiler intelligently chooses the best way to translate a problem formulation and choice of optimization algorithm into an efficient solver implementation.

Face2Face: Real-time Face Capture and Reenactment of RGB Videos

Justus Thies, Michael Zollhöfer, Marc Stamminger, Christian Theobalt, Matthias Nießner

CVPR 2016 (Oral)

We present a novel approach for real-time facial reenactment of a monocular target video sequence (e.g., Youtube video). The source sequence is also a monocular video stream, captured live with a commodity webcam. Our goal is to animate the facial expressions of the target video by a source actor and re-render the manipulated output video in a photo-realistic fashion.

Volumetric and Multi-View CNNs for Object Classification on 3D Data

Charles Ruizhongtai Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, Leonidas Guibas

CVPR 2016 (Spotlight)

In this paper, we improve both Volumetric CNNs and Multi-view CNNs by introducing new distinct network architectures. Overall, we are able to outperform current state-of-the-art methods for both Volumetric CNNs and Multi-view CNNs.

Real-time Expression Transfer for Facial Reenactment

Justus Thies, Michael Zollhöfer, Matthias Nießner, Levi Valgaerts, Marc Stamminger, Christian Theobalt

ACM Transactions on Graphics 2015 (TOG)

We present a method for the real-time transfer of facial expressions from an actor in a source video to an actor in a target video, thus enabling the ad-hoc control of the facial expressions of the target actor.

Activity-centric Scene Synthesis for Functional 3D Scene Modeling

Matthew Fisher, Manolis Savva, Yangyan Li, Pat Hanrahan, Matthias Nießner

ACM Transactions on Graphics 2015 (TOG)

We present a novel method to generate 3D scenes that allow the same activities as real environments captured through noisy and incomplete 3D scans.

SemanticPaint: Interactive 3D Labeling and Learning at your Fingertips

Julien Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, Philip H. S. Torr

ACM Transactions on Graphics 2015 (TOG)

We present a new interactive and online approach to 3D scene understanding. Our system, SemanticPaint, allows users to simultaneously scan their environment, while interactively segmenting the scene simply by reaching out and touching any desired object or surface.

Efficient Ray Tracing of Subdivision Surfaces using Tessellation Caching

Carsten Benthin, Sven Woop, Matthias Nießner, Kai Selgrad, Ingo Wald

High Performance Graphics 2015

In this paper, we propose method to efficiently ray trace subdivision surfaces using a lazy-build caching scheme while exploiting the capabilities of today's many-core architectures. Our approach is part of Intel's Embree.

Real-time Rendering Techniques with Hardware Tessellation

Matthias Nießner, Benjamin Keinert, Matthew Fisher, Marc Stamminger, Charles Loop, Henry Schäfer

Computer Graphics Forum 2015

In this survey, we provide an overview of real-time rendering techniques with hardware tessellation by summarizing, discussing, and comparing state-of-the art approaches.

Shading-based Refinement on Volumetric Signed Distance Functions

Michael Zollhöfer, Angela Dai, Matthias Innmann, Chenglei Wu, Marc Stamminger, Christian Theobalt, Matthias Nießner

ACM Transactions on Graphics 2015 (TOG)

We present a novel method to obtain fine-scale detail in 3D reconstructions generated with low-budget RGB-D cameras or other commodity scanning devices.

Exploiting Uncertainty in Regression Forests for Accurate Camera Relocalization

Julien Valentin, Matthias Nießner, Jamie Shotton, Andrew Fitzgibbon, Shahram Izadi, Philip H. S. Torr

CVPR 2015

In this paper, we train a regression forest to predict mixtures of anisotropic 3D Gaussians and show how the predicted uncertainties can be taken into account for continuous pose optimization. Experiments show that our method is able to relocalize up to 40 percent more frames than the state of the art.

Incremental Dense Semantic Stereo Fusion for Large-Scale Semantic Scene Reconstruction

Vibhav Vineet, Ondrej Miksik, Morten Lidegaard, Matthias Nießner, Stuart Golodetz, Victor A. Prisacariu, Olaf Kähler, David W. Murray, Shahram Izadi, Patrick Pérez, Philip H. S. Torr

ICRA 2015

In this paper, we build on a recent hash-based technique for large-scale fusion and an efficient mean-field inference algorithm for densely-connected CRFs to present what to our knowledge is the first system that can perform dense, large-scale, outdoor semantic reconstruction of a scene in (near) real time.

The Semantic Paintbrush: Interactive 3D Mapping and Recognition in Large Outdoor Spaces

Ondrej Miksik, Vibhav Vineet, Morten Lidegaard, Ram Prasaath, Matthias Nießner, Stuart Golodetz, Stephen L. Hicks, Patrick Pérez, Shahram Izadi, Philip H. S. Torr

CHI 2015

We present an augmented reality system for large scale 3D reconstruction and recognition in outdoor scenes. Unlike existing prior work, which tries to reconstruct scenes using active depth cameras, we use a purely passive stereo setup, allowing for outdoor use and extended sensing range.

Database-Assisted Object Retrieval for Real-Time 3D Reconstruction

Yangyan Li, Angela Dai, Leonidas Guibas, Matthias Nießner

Eurographics 2015

We present a novel reconstruction approach based on retrieving objects from a 3D shape database while scanning an environment in real-time. With this approach, we are able to replace scanned RGB-D data with complete, hand-modeled objects from shape databases.

Dynamic Feature-Adaptive Subdivision

Henry Schäfer, Jens Raab, Mark Meyer, Marc Stamminger, Matthias Nießner

I3D 2015

In this paper, we present dynamic feature-adaptive subdivision (DFAS), which improves upon FAS by enabling an independent subdivision depth for every irregularity.

SceneGrok: Inferring Action Maps in 3D Environments

Manolis Savva, Angel X. Chang, Pat Hanrahan, Matthew Fisher, Matthias Nießner

ACM Transactions on Graphics 2014 (TOG)

In this paper, we present a method to establish a correlation between the geometry and the functionality of 3D environments.

Real-time Shading-based Refinement for Consumer Depth Cameras

Chenglei Wu, Michael Zollhöfer, Matthias Nießner, Marc Stamminger, Shahram Izadi, Christian Theobalt

ACM Transactions on Graphics 2014 (TOG)

We present the first real-time method for refinement of depth data using shape-from-shading in general uncontrolled scenes.

Real-time Non-rigid Reconstruction using an RGB-D Camera

Michael Zollhöfer, Matthias Nießner, Shahram Izadi, Christoph Rhemann, Christopher Zach, Matthew Fisher, Chenglei Wu, Andrew Fitzgibbon, Charles Loop, Christian Theobalt, Marc Stamminger

ACM Transactions on Graphics 2014 (TOG)

We present a combined hardware and software solution for markerless reconstruction of non-rigidly deforming physical objects with arbitrary shape in real-time.

Real-Time Deformation of Subdivision Surfaces from Object Collisions

Henry Schäfer, Benjamin Keinert, Matthias Nießner, Christoph Buchenau, Michael Guthe, Marc Stamminger

High Performance Graphics 2014

We present a novel real-time approach for fine-scale surface deformations resulting from collisions.

Local Painting and Deformation of Meshes on the GPU

Henry Schäfer, Benjamin Keinert, Matthias Nießner, Marc Stamminger

Computer Graphics Forum 2014

We present a novel method to adaptively apply modifications to scene data stored in GPU memory. Such modifications

RetroDepth: 3D Silhouette Sensing for High-Precision Input On and Above Physical Surfaces

David Kim, Shahram Izadi, Jakub Dostal, Christoph Rhemann, Cem Keskin, Christopher Zach, Jamie Shotton, Tim Large, Steven Bathiche, Matthias Nießner, Alex Butler, Sean Fanello, Vivek Pradeep

CHI 2014

We present RetroDepth, a new vision-based system for accurately sensing the 3D silhouettes of hands, styluses, and other objects, as they interact on and above physical surfaces.

State of the Art Report on Real-time Rendering with Hardware Tessellation

Henry Schäfer, Matthias Nießner, Benjamin Keinert, Marc Stamminger, Charles Loop

Eurographics 2014

In this state of the art report, we provide an overview of recent work and challenges in this topic by summarizing, discussing and comparing methods for the rendering of smooth and highly detailed surfaces in real-time.

Combining Inertial Navigation and ICP for Real-time 3D Surface Reconstruction

Matthias Nießner, Angela Dai, Matthew Fisher

Eurographics 2014

We present a novel method to improve the robustness of real-time 3D surface reconstruction by incorporating inertial sensor data when determining inter-frame alignment. With commodity inertial sensors, we can significantly reduce the number of iterative closest point (ICP) iterations required per frame.

Real-time 3D Reconstruction at Scale using Voxel Hashing

Matthias Nießner, Michael Zollhöfer, Shahram Izadi, Marc Stamminger

ACM Transactions on Graphics 2013 (TOG)

Online 3D reconstruction is gaining newfound interest due to the availability of real-time consumer depth cameras. We contribute an online system for large and fine scale volumetric reconstruction based on a memory and speed efficient data structure.

Analytic Displacement Mapping using Hardware Tessellation

Matthias Nießner, Charles Loop

ACM Transactions on Graphics 2013 (TOG)

Displacement mapping is ideal for modern GPUs since it enables high-frequency geometric surface detail on models with low memory I/O. We provide a comprehensive solution to these problems by introducing a smooth analytic displacement function.

Real-time Collision Detection for Dynamic Hardware Tessellated Objects

Matthias Nießner, Christian Siegl, Henry Schäfer, Charles Loop

Eurographics 2013

We present a novel method for real-time collision detection of patch based, displacement mapped objects using hardware tessellation. Our method supports fully animated, dynamically tessellated objects and runs entirely on the GPU.

Rendering Subdivision Surfaces using Hardware Tessellation

Matthias Nießner

PhD Thesis (Published by Dr. Hut)

In this thesis we present techniques that facilitate the use of high-quality movie content in real-time applications that run on commodity desktop computers. We utilize modern graphics hardware and use hardware tessellation to generate surface geometry on-the-fly based on patches.

Real-time Simulation of Human Vision using Temporal Compositing with CUDA on the GPU

Matthias Nießner, Nadine Kuhnert, Kai Selgrad, Marc Stamminger, Georg Michelson

Proceedings 25th Workshop on Parallel Systems and Algorithms 2013

We present a novel approach that simulates human vision including visual defects such as glaucoma by temporal composition of human vision in real-time on the GPU. Therefore, we determine eye focus points every time step and adapt the lens accommodation of our virtual eye model accordingly.

Feature Adaptive GPU Rendering of Catmull-Clark Subdivision Surfaces

Matthias Nießner, Charles Loop, Mark Meyer, Tony DeRose

ACM Transactions on Graphics 2012 (TOG)

We present a novel method for high-performance GPU based rendering of Catmull-Clark subdivision surfaces. Unlike previous methods, our algorithm computes the true limit surface up to machine precision, and is capable of rendering surfaces that conform to the full RenderMan specification for Catmull-Clark surfaces.

Patch-based Occlusion Culling for Hardware Tessellation

Matthias Nießner, Charles Loop

Computer Graphics International 2012

We present an occlusion culling algorithm that leverages the unique characteristics of patch primitives within the hardware tessellation pipeline.

Efficient Evaluation of Semi-Smooth Creases in Catmull-Clark Subdivision Surfaces

Matthias Nießner, Charles Loop, Günther Greiner

Eurographics 2012

We present a novel method to evaluate semi-smooth creases in Catmull-Clark subdivision surfaces. Our algorithm supports both integer and fractional crease tags corresponding to the RenderMan (Pixar) specification.

Real-time Simulation and Visualization of Human Vision through Eyeglasses on the GPU

Matthias Nießner, Roman Sturm, Günther Greiner

ACM SIGGRAPH VRCAI 2012

We present a novel approach that allows real-time simulation of human vision through eyeglasses. Our system supports glasses that are composed of a combination of spheric, toric and in particular of free-form surfaces.

Effective Back-Patch Culling for Hardware Tessellation

Charles Loop, Matthias Nießner, Christian Eisenacher

Proceedings of Vision, Modeling, and Visualization 2011

When rendering objects with hardware tessellation, back-facing patches should be culled as early as possible to avoid unnecessary surface evaluations, and setup costs for the tessellator and rasterizer.

Fast Indirect Illumination Using Layered Depth Images

Matthias Nießner, Henry Schäfer, Marc Stamminger

Computer Graphics International 2010

We present a novel hybrid rendering method for diffuse and glossy indirect illumination.