Light fields and computational photography


(a) (b) (c) (d) (e) (f) (g) (h)
(i) (j) (k) (l) (m) (n) (o)

Marc Levoy has retired from Stanford University to lead a team at Google. This project is no longer active as a Stanford research project. However, many of the technologies developed during this project live on in products such as Google StreetView (e), the Lytro Camera (j), and Google's Jump camera (o).


Overview

The light field, first described in Arun Gershun's classic 1936 paper of the same name, is defined as radiance as a function of position and direction in regions of space free of occluders. In free space, the light field is a 4D function - scalar or vector depending on the exact definition employed. Light fields were introduced into computer graphics in 1996 by Marc Levoy and Pat Hanrahan. Their proposed application was image-based-rendering - computing new views of a scene from pre-existing views without the need for scene geometry. (A workshop on image-based modeling and rendering was held at Stanford in 1998.)

Since 1996, research on light fields has followed a number of lines. On the theoretical side, researchers have developed spatial and frequency domain analyses of light field sampling and have proposed several new parameterizations of the light field, including surface light fields and unstructured Lumigraphs. On the practical side, researchers have experimented with literally dozens of ways to capture light fields, ranging from camera arrays to kaleidoscopes, as well as several ways to display them, such as an array of video projectors aimed at a lenticular sheet. Researchers have also explored the relationship between light fields and other sampled representations of light transport, such as incident light fields and reflectance fields. At Stanford, we have focused on the boundary between light fields, photography, and high-performance imaging, an area we sometimes call computational photography. (A workshop on this theme was held at MIT in May of 2005.) However, computational photography has grown to become broader than light fields, and our research also touches on other aspects of light fields, such as interactive animation of light fields and computing shape from light fields.

The images above depict some of the work we have done in our laboratory on light fields:

(a) An image from our SIGGRAPH 1996 paper, showing an array of renderings of a 3D computer model of a buddha statuette (at top) and the transpose of the light field (at bottom).
(b) The Stanford Spherical Gantry, a multi-purpose 4-degree of freedom motorized gantry we built for capturing light fields and bidirectional reflectance distribution functions (BRDFs) of small stationary objects.
(c) We first demonstrated synthetic focusing of light fields in 1996. Here is our first continuous focusing, from 2002. The array of input images was captured by the same robot/camera rig used to capture a light field of Michelangelo's statue of Night in 1999. Focusing was done in software, producing this movie.
(d) An array of 128 synchronized CMOS video cameras we built as part of the Stanford Multi-Camera Array project.
(e) PhD student Gaurav Garg rides in a pickup truck with a sideways-looking video camera, which we used to construct multi-perspective panoramas of urban landscapes in the Stanford CityBlock project. Google's StreetView grew out of this project.
(f) An array of 16 planar mirrors, into which we aimed a high-resolution camera and projector to produce 16 virtual cameras and projectors for our paper on synthetic aperture confocal imaging.
(g) An array of curved mirrorlets, created by machining and chrome-plating an aluminum block. Arrays like this are an alternative to aiming a camera into an array of microlenses.
(h) A photo-resister, which we used in conjunction with a video projector to capture slices of a light field using a technique called dual photography.
(i) PhD student Ren Ng holds a medium format SLR camera into which he has inserted a microlens array between the sensor and main lens. The camera captures light fields instead of photographs. Here are papers describing the camera and its theory of operation. Ren's PhD dissertation, "Digital Light Field Photography," won the 2006 ACM Doctoral dissertation Award.
(j) Ren is founder and current CEO of Lytro, a startup company that is commercializing his dissertation research. Pictured here is their camera, which captures a light field you can synthetically focus (a.k.a. digitally refocus) after capture.
(k) By inserting a microlens array (circled in red) into a standard microscope, we create a light field microscope (LFM). From its images we can produce perspective flybys, focal stacks, and volume datasets at a single instant in time. Here is a paper describing the idea.
(l) By inserting a second microlens array and video projector into the light path of the same microscope, we can control the incident light field falling on a specimen as well as the light field leaving it. In this example, we use our Light Field Illuminator (LFI) to change the characteristics of light falling on a single blond human hair. Here is a paper describing the idea.
(m) We sometimes create light fields synthetically. This Venn diagram shows a taxonomy of the kinds of apertures that can occur in light fields, under a formulation we have devised of general linear cameras with finite (i.e. non-pinhole) apertures.
(n) In another theoretical paper, we explore the relationship between light fields as used in the computer graphics community and the Wigner distribution commonly used in the wave optics community.
(o) A ring of 16 synchronized Go-Pro cameras, built by Google as part of its Jump system and announced in May 2015 at Google I/O. Although not a product of our laboratory, this is a light field camera. Levoy's team at Google advised on its design and contributed software to the system.

Recent papers in this area:

Gyro-Based Multi-Image Deconvolution for Removing Handshake Blur
Sung Hee Park, Marc Levoy
Proc. CVPR 2014
WYSIWYG Computational Photography via Viewfinder Editing
Jongmin Baek, Dawid PajÄ…k, Kihwan Kim, Kari Pulli, Marc Levoy
Proc. SIGGRAPH Asia 2013
Unstructured Light Fields
Abe Davis, Fredo Durand, Marc Levoy
Proc. Eurographics 2012
Applications of Multi-Bucket Sensors to Computational Photography
Gordon Wan, Mark Horowitz Marc Levoy
Stanford Computer Graphics Laboratory Technical Report 2012-2
Focal stack compositing for depth of field control
David E. Jacobs, Jongmin Baek, Marc Levoy
Stanford Computer Graphics Laboratory Technical Report 2012-1
Digital Video Stabilization and Rolling Shutter Correction using Gyroscopes
Alexandre Karpenko, David E. Jacobs, Jongmin Baek, Marc Levoy
Stanford Computer Science Tech Report CSTR 2011-03
Wigner Distributions and How They Relate to the Light Field
Zhengyun Zhang, Marc Levoy
IEEE International Conference on Computational Photography (ICCP) 2009
Veiling Glare in High Dynamic Range Imaging
Eino-Ville (Eddy), Andrew Adams, Mark Horowitz, Marc Levoy
Proc. SIGGRAPH 2007
General Linear Cameras with Finite Aperture
Andrew Adams, Marc Levoy
Eurographics Symposium on Rendering (EGSR) 2007
Light Fields and Computational Imaging
Marc Levoy
IEEE Computer, August 2006
Symmetric Photography: Exploiting Data-sparseness in Reflectance Fields
Gaurav Garg, Eino-Ville (Eddy) Talvala, Marc Levoy, Hendrik P.A. Lensch
Proc. 2006 Eurographics Symposium on Rendering
Light Field Photography with a Hand-Held Plenoptic Camera
Ren Ng, Marc Levoy, Mathieu Brédif, Gene Duval, Mark Horowitz, Pat Hanrahan
Stanford University Computer Science Tech Report CSTR 2005-02
Fourier Slice Photography
Ren Ng
Proc. SIGGRAPH 2005
Dual Photography
Pradeep Sen, Billy Chen, Gaurav Garg, Steve Marschner, Mark Horowitz, Marc Levoy, Hendrik Lensch
Proc. SIGGRAPH 2005
Interactive Deformation of Light Fields
Billy Chen, Eyal Ofek, Harry Shum, Marc Levoy
Proc. Symposium on Interactive 3D Graphics and Games (I3D) 2005
Synthetic aperture confocal imaging
Marc Levoy, Billy Chen, Vaibhav Vaish, Mark Horowitz, Ian McDowall, Mark Bolas
Proc. SIGGRAPH 2004
See also the papers produced by the Stanford Multi-Camera Array, Light Field Microscope, and CityBlock projects.

Some older papers:

Light Field Rendering
Marc Levoy and Pat Hanrahan,
Proc. SIGGRAPH '96

Slides from talks:

(Listed in reverse chronological order. (Slides from papers may also be available on the web pages of those papers.)

Demos:

Available software and data:

Publicity about the project

Financial support:


If images on this page look dark to you, see our note about gamma correction.

A list of technical papers, with abstracts and pointers to additional information, is also available. Or you can return to the research projects page or our home page.


webmaster@graphics.stanford.edu