CS 448A - Computional photography

Ideas for projects

Many of these project ideas could be Siggraph or CVPR/ICCV papers. In fact, some of the projects in this course will probably end up being submitted for publication! To keep you and your fellow students from being scooped, please don't distribute this handout, or the URL of this web page, or its ideas, beyond the Stanford community. At the end of each project idea is the author of that paragraph (Marc or Hendrik); they might be the best person to chat with about that project.

High-performance imaging using the Stanford Multi-Camera Array

In our CVPR and SIGGRAPH papers we explored several arrangements of cameras: tightly packed, widely spaced, single line, etc. One arrangement we never tried was a circle or sphere of cameras, with the cameras looking inward towards the center. Can you fly around and towards an object - Matrix-style - using the images collected by such an array? Can you compute synthetic aperture photographs from them? How about so-called "peripheral panoramas? What computer vision algorithms (shape-from-stereo, shape-from-focus, etc.) work well in such an arrangement? (Marc)

Our tiled arrangement of cameras with telephoto lenses has the unique ability to capture each portion of a field of view using a different exposure, but despite figuring prominently in our SIGGRAPH 2005 paper, we never came up with a good automatic exposure metering algorithm for the array. Specifically, given an initial panorama captured by the array, or a low-resolution photograph captured by a separate wide-angle camera, what is the best exposure setting for each of the 100 cameras? And if you free-run the cameras, producing a video stream, what are good dynamic automatic exposure metering algorithms? (Marc)

If our purpose in building an array of cameras was to explore "high-X imaging", one of the "X" we never got to was multi-spectral imaging. Using an array of cameras and a sequence of narrow-passband color filters, one can acquire a collection of images that discretely sample the visible spectrum (400nm \- 700nm). Potential applications include spectral sampling for computer graphics - the ability to capture a scene under broadband illumination, then digitally relight it later using colored lights), and the ability to see through camouflage. We have the cameras and the filters; we've just never tried the experiment. (Marc)

Determining the 3D shape of an object from multiple images is one of the central problems in computer vision, and many algorithms have been proposed for it. Having more images generally improves the performance of these algorithms, so given 100 cameras, we should do well. But which algorithms benefit most from having more images? Try one, off the shelf or roll your own. How about shape-from-focus using synthetically focused images? Can you think up a new algorithm for the "shape from light field" problem? (Marc)

Other high-performance imaging projects

Imagine a camera in which images are captured continuously during aiming and focusing. Could this extra imagery be used to boost the resolution, dynamic range, or other aspects of the snapshot taken when the photographer presses the shutter? Try it. Use image blending (Sawhney, Siggraph 2001), texture synthesis (Wei and Levoy, Siggraph 2000), image analogies (Hertzmann, Siggraph 2001), or an algorithm of your own invention. Scaling this idea up, if you seed an array of wide-angle cameras with a few telephoto cameras, which capture bits of the same field of view but at higher resolution, can you use this information to create a wide-angle, high-resolution panorama? (Marc)

Imagine a camera that sweeps quickly through a range of focus settings, taking an image at every setting. This would allow you to select the focus and depth of field afterwards using an interactive graphics program. Design and implement such a program. What additional creative effects can you achieve using this technique? For this project, mount your camera on a tripod to avoid the need to first stabilize the imagery. (Marc)

Combining multiple images

Image stabilization, which is a mature technology in video cameras, could have many uses in still photography. Implement an image stabilization algorithm from the vision literature, then use it to stabilize and combine a set of images shot using a handheld camera. This should allow you to shorten exposure times in low-light scenes without enhancing noise, and to produce cleaner shadows in high-contrast scenes. Using non-linear image combinations, you might be able to remove cars from a freeway, or crowds from a plaza. (Marc)

Combining a set of photographs taken under different conditions, such as at different times, or under different exposure, focus, or illumination (e.g. with and without flash) has become a major theme at Siggraph, but we've surely barely scratched the surface in this area. Look at the Digital Photomontage paper (Agarwala, Siggraph 2004) or the several flash-noflash papers (Petschnigg, SIggraph 2004, Raskar, Siggraph 2004, Agrawal, Siggraph 2005), then think up something they haven't tried. Petschnigg's paper began with a project by him in this course four years ago! (Marc)

View interpolation using image statistics. Fitzgibbon et al. (ICCV 2003) proposed a method for image-based rendering using image-based priors where view interpolation of objects of even complex geometry results in decent images by ensuring that every pixel in a reconstructed view is somehow probable in the context of its reconstructed neighbor pixels. The distribution of reasonable neighborhoods is derived directly from the input images. Implement this technique and extend it by applying acceleration techniques found in current texture synthesis papers. (Hendrik)

Image statistics for refocusing. In a series of photographs of a scene with a hand-held point-and-shoot camera there are typically one or two images which captured the correct moment in time but are just out of focus while others, more boring shots might be perfectly sharp. By using image statistics (see above), i.e. knowledge about the distribution of pixel neighborhoods, derived from the sharp images it should be possible to derive a refocused image for the point in time you are actually interested in. (Hendrik)

Video enhancement by space-time filtering and motion compensation. Videos captured in low-light conditions typically are underexposed or rather noisy. By applying an adaptive filter which averages over time for static pixels and averages over neighboring pixels within the same frame for moving pixels Bennett and McMillan (SIGGRAPH 2005) were able to enhance and tonemap low dynamic range videos to decent videos at decent exposure. While their current filter is oriented to either the time or the spatial domain better result might be achieved by performing motion compensation, i.e. orienting the filter along some curve in the space-time continuum. (Hendrik)

Projects involving projectors

Suppose you have a scene containing one or more foreground objects, an array of projectors, and a set of per-projector mattes that identify for each projector those pixels belonging to the foreground versus the background. Try shaping the illumination that falls on the scene in interesting ways. For example, illuminate one foreground object without illuminating the background or casting shadows on the ground, or illuminate the background while leaving the foreground dark. There are several easy ways to compute the mattes. Imagine the theatrical applications! (Marc)

In our Siggraph 2004 paper on confocal imaging, we explored only a few applications of shaped illumination; there must be many others. Try using it to cloak an object, i.e. make it disappear. Try shaping the illumination of a sculpture to change its apparant shape or surface properties, or the illumination in an office to create a virtual whiteboard on your bookcase, or a virtual library on your whiteboard. Create (or simulate) adaptively shaped headlights for a car or bicycle. (Marc)

Projects involving projectors and cameras

By projecting synthetically focused textures onto an object, existing shape-from-focus algorithms should perform better. In fact, there are several kinds of texture-enhanced vision algorithms that could benefit from having an array of projectors, an array of cameras, or both. Try implementing one of these algorithms. (Marc)

By displaying temporal stripe sequences on two projectors having overlapping fields of view, one can project a unique time code onto every point in 3-space. Given such a volumetric coding of space, a photocell placed anywhere in a room can figure out where it is. Alternatively, one could build a powerful rangefinder using these two projectors and any number of uncalibrated cameras - to see around occlusions. To our knowledge, neither of these ideas has been published. (Marc)

3D scanning with multiple projectors. 3D scanning with a single projector is an almost solved problem. However, the number of pixels seen by the camera which cannot be reliably illuminated by a single projector is typically significant due to occlusion. Using multiple projectors (more than the two called for in the previous project) it should be possible to illuminate many more surface points even for highly occluded scenes. Unfortunately, the light patterns typically used in a single-projector setup turn out to be not suitable since multiple projector might illuminate the same surface points leading to disturbed readings. Design a 3D scanning setup where multiple projectors illuminate the scene at the same time (projector and camera calibration code is given). (Hendrik)

3D scanning of refracting objects. Capturing the environment matte of a glass objects allows to reproduce the appearance of the object for arbitrary planar backgrounds. For refracting objects environment matting typically yields a one-to-one mapping from camera pixels to locations on a monitor where the viewing ray after interacting with the object would intersect the monitor plane. Capturing environment mattes from multiple view points, it should be possible to reconstruct the 3D shape of refracting objects, at least for those objects with relatively smooth geometry. (Hendrik)

Purely analytical or design projects

An array of cameras can be treated as a gigasample-per-second photometer, with wide latitude over how these samples are distributed in space, time, and wavelength. Researchers have investigated alternative pixel lattices for cameras, but nobody has thought about lattices of camera viewpoints, nor the general problem of arranging samples in space, time, dynamic range, focus, and wavelength. One can imagine sampling strategies that are optimal for a given scene or class of scenes, or strategies that are adaptive. This is a wide-open problem; in fact, it's a good dissertation topic. (Marc)

Analyze the applicability of an exotic or computational imaging technique from microscopy, astronomy, or another field to photography, i.e. at the human scale using combinations of cameras, projectors, or other optical components. An example from microscopy might be phase contrast illumination; an example from astronomy might be coded-aperture imaging (using an array of pinholes on a mask). (Marc)

Object cloaking with markers. Given a projector and a camera one could create an augmented-reality book, i.e. a white plane in the real world onto which a projector projects a texture which stays fixed even for arbitrary orientation of the book. Tracking the pose of a single plane using a video camera it should be easy to compute the corresponding distortion which is necessary to display an image that stays always at the same place relative to the real plane. (Hendrik)

Super-resolution in reflectance fields. Assume you are given two 4D reflectance fields of a scene from two projectors and a single camera, and in addition an individual picture of the scene illuminated by a single point light source positioned in between the projectors. What techniques could be used to derive the 4D reflectance field from a virtual projector at the location of the point light source? Transfer techniques for computing super-resolution in image and videos to the light domain. (Hendrik)

And a purely software project

If the future of photography includes multiple images, then image editing programs should support arrays of images as a fundamental datatype, so that one can manipulate panoramic image sets, multi-exposure images, video sequences, light fields, and so on. Such a Photoshop++ program should also support images of arbitrary range dimensionality, such as rgbaz (range images), bump or normal maps, shader parameter maps, etc. Finally, it should offer built-in operators that are appropriate for those datatypes, e.g. differential geometry operators for range images, lighting operators for bump maps and BRDFs, etc. Anybody want to try writing such a program? (Marc)