Projects
Real-time 3D Reconstruction for Static and Dynamic Scenes
Recent advances commodity 3D sensors (e.g., Microsoft Kinect), and their ensuing widespread avilability, has inspired significant interest in using this sensing technology for mapping and understanding 3D environments. In fact, the efficient capture of 3D environments – in particular, reconstructing the real world in an interactive context – is crucial to enabling extensive digitization of real-world scenes as well as semantic scene understanding. There are several key challenges to real-time reconstruction of 3d enviroments: a scene representation facilitating scalability; robust and accurate global tracking; handling dynamic events. To this end, we have developed a method for obtaining 3D reconstructions of large-scale (static) scenes in real-time, producing simultaneous localization and mapping of the real world.
However, real-world enviroments are usually not static, with people or objects moving about. We have developed an approach to reconstruct and track dynamic objects in an unconstrained interactive context – the first to the capture of dynamic geometry with a commodity depth sensor in real time.
Interactive Video Manipulation: Photoshopping Videos in Real Time
While PhotoShop has made image manipulation easy and often unnoticeable to the human eye, video manipulation proves a more challenging task, requiring temporal coherency and realistic movement and flow. In order to achieve photo-realistic video editing, we first reconstruct the underlying 3D geometry (and material properties), and manpulate the video through its geometry. Currently, we focus on human faces, which we accurately reconstruct and track, enabling the transfer of facial expressions from a source actor to a target actor. Our paper received wide media attention, including coverage by New York Times, Wall Street Journal, The Verge, and more :)
 |
 |
Real-time Non-Linear Optimization: Domain Specific Language
Much of our research has involved a significant amount of effort hand-crafting individual GPU-based solvers for various non-linear optimization problems, in order to enable real-time performance (e.g., dynamic reconstruction, shape-from-shading, global pose alignment). As such non-linear optimization problems are prevalent in computer graphics and vision, our goal is to create a domain specific language which makes it easy to solve these problems at high performance, abstracting away the need for expertise on both modern GPUs and optimization.
Subdivision Surface Rendering: OpenSubdiv
The geometric properties and high surface quality of subdivision surfaces has led to the adoption of subdivision surfaces as a movie industry standard for modeling 3D geometry. However, the computation required to evaluate these surfaces is non-trivial; our feature-adaptive subdivision method provides exact computation of the surface (up to machine precision) at very high performance – accelerating the evaluation of subdivision surfaces by several orders of magnitude. This enables the rendering of high-quality movie content on commercially available desktop computers within only a few milliseconds, opening up the possibility of using subdivision surfaces in real-time applications such as video games. The results of this work have been made publicly available in Pixar's open source initiative OpenSubdiv, which is used in many industry applications (e.g., Maya). Our algorithms can also be found in video games such as Activision's best seller Call of Duty: Ghosts, where we now can enjoy the surface quality of movies.
 |
 |