Revision 4 as of 2007-02-14 19:40:13

    Assignment5

Assignment 5 - High Dynamic Range Photography

Due Date: Thursday February 22nd, 11:59PM

Questions? Check out the Assignment 5 FAQ and discussion page.

In this assignment you'll explore high dynamic range photography. You'll take your own photos, generate a high dynamic range image from them, and explore a couple of ways of displaying HDR images on low dynamic range displays. There are multiple steps to this assignment, but they are essentially independent. We list the steps in the logical order, but you may want to do later, simpler steps first to help your understanding.

Steps

Download and build the starter code

  1. Begin by downloading the Assigment 5 starter code and latest version of the libST source here. Note that you must use the new version of libST, it has been updated since the previous assignment.

  2. Build libST. Building libST should require the same steps on your development system as it did in previous assignments. A number of changes have been made to the code which are significant for this assignment. See the descriptions below for the full details of the changes, which you should look over and understand at a high level.
  3. Build the assignment 5 starter code. The subdirectory /assignment5 contains the starting code for your project. This directory should contain the C++ source file hdr.cpp. You will provide us your modified version of hdr.cpp as part of this assignment's handin. The directory also response.h, response.cpp, and some support code in the directory /tnt. You should understand the interface provided in response.h and response.cpp since you will use it in your code. You are not responsible for any of the /tnt code. Finally, we've provided d40.cr, a response curve file for the Nikon D40. If you use our cameras this file will work, but if you have your own camera you must generate your own camera response curve. See the instructions below for how to do that.

4. If you want to start working on the assignment before you get to take your own photos, download the sample photos and hdr images from here. Note that part of the assignment, and your grade, is taking your own photos. We are only providing these so you can get started sooner.

Understanding Changes to libST

There have been a few changes to libST, but you are already familiar with much of the code.

  • STColor - This class has been updated with many overloaded operators and a couple of helper functions. Be familiar with what's available here. This is the representation we use for pixels in an HDR image, so most of your code will deal with this class.
  • STHDRImage - This class is analogous to the STImage class and has almost equivalent functionality. The main difference is that each pixel is an STColor. STColors are floating point and can take on any valid float value. However, when displayed, they will be clamped to [0, 1]. We've added an operator()(int x, int y) so accessing pixels uses the simple notation hdrimage(x,y) (which returns an STColor&. Note the same operator has been implemented for STImage.

Review of Camera Imaging and HDR

The input to a camera is the radiance of the scene. Light enters the camera, passes through the lens system and aperature producing some irradiance Ei at each pixel i. The camera can control how much energy actually hits the sensors by increasing/decreasing the shutter speed. By integrating the irradiance Ei over this shutter time dt we get the sensor exposure, Xi. The many steps from this point to producing an actual image (film exposure, development, digital conversion, remapping) act as a non-linear function which maps these input exposures in the range [0,infinity], to image values Zi in the range [0,255]. This function f(Xi) = Zi is called the camera response curve.

Our goal is to use some output images to get the input Xi values. Therefore, we actually want to consider the inverse function f-1(Zi) = Xi, and by substitution f-1(Zi) = Ei*dt. Taking the natural logarithm we have ln(f^-1(Zi)) = ln(Ei) + ln(dt). We'll rename this function g(Zi) = ln(Ei) + ln(dt). This is the representation we'll actually use for the response curve (given a pixel value, what is the ln(exposure) ). Note that this represents exactly the same information, we've just adjusted the representation a bit. This is convenient for our purposes because different shutter speeds are equivalent to translations of this function.

Here we show a sample response curve. XXX Image

Answer These Questions

(Please include answers to the following with your handin)

Taking Photographs

Two D40's, available 2 hours at a time, signup

What to take pictures of, how to use the camera, # photos, range of shutter speeds

Creating an HDR Image

An HDR image is just the irradiance values Ei at each pixel due to the scene. In other words, an HDR image represents the actual power incident on each sensor, before any of the other effects of the camera after the lens and aperature, have taken place. Assume we have the g(Zi) representation of the response curve and just a single input image with known shutter speed dt. Then we could simply rearrange the response curve formula to find Ei:

g(Zi) = ln(Ei) + ln(dt)

ln(Ei) = g(Zi) - ln(dt)

Ei = exp( g(Zi) - ln(dt) )

So why can't we just use a single photograph? The response curve can't give pixel values less than 0 or greater than 255, so all exposures above a certain level will result in a response of 255 and similarly for exposures below a certain level. This means that multiple exposure values map to the same response pixel value, so the function cannot actually be inverted. Therefore the above equations cannot determine the actual exposure value if the input pixel value is too small or too large.

So we need enough photographs such that every pixel is within a usable range in at least one photo (best if its, e.g., in [64,192]). Using this method, j photos will result in j exposure estimates for each pixel (that is, we have Eij = exp( g(Zij) - ln(dt_j) ) for each pixel i in each image j). Then we need to combine them somehow. To do this we'll weight them so that Eij will have high weight if pixel Zij has a high confidence value, i.e. those in the middle of the response curve. We've already created a weight function for you (CameraResponse::Weight()) which accomplishes this. Then we just calculate a weighted average (which we do on the ln values, not the actual values):

ln (Ei) = sum_over_j( weight(Zij)*(g(Zij) - ln(dt_j)) ) / sum_over_j( weight(Zij) )

and the Ei can easily be found.

You should implement this function in recover_hdr(). Only the arguments to the function are necessary to accomplish this. Carefully read the comments for the functions of the CameraResponse class to make sure you're getting the values you expect (i.e. whether you're getting the value itself or ln(value)).

You can perform this process by running

hdr -create photos.list response.cr photo_out.pfm

which will take the file photos.list and the response curve response.cr and create the HDR image and save it to photo_out.pfm. Note that pfm files are uncompressed, so for high resolution images such as those from cameras, the file will be quite large (close to 100 megabytes).

Taking a Virtual Photograph

Given an HDR image and a camera response we can actually find the image that would be generated by that camera. This is a straightforward use of the response curve. For some shutter speed dt, we find the exposure for each pixel and look up the response. We've given you a function in CameraResponse to lookup the response for some exposure, so this should be simple to implement.

The camera response curve we use doesn't necessarily need to be the one the original photographs were taken with. You can actually find out what the same image, taken with different imaging systems, would look like!

Implement this process in the function virtual_photo(). When you run the program with

hdr -vp photo.pfm response.cr

it will display calculate and display virtual photographs. You can increase/decrease the shutter speed using +/-. It increments in 1/3 stops.

Displaying an HDR Image: Linear Mapping

In order to display an HDR image we need to map it to a low dynamic range display. Linear mapping is the simplest approach and simply scales and clamps the HDR values so a certain range is displayed and any values outside that range are clamped to the maximum LDR value. Implement this approach in scale_hdr so that [0,max_val] in the HDR image maps to [0,255] in the LDR image.

When you run the program with

hdr -view photo.pfm

it will use this display scheme. Use +/- to increase/decrease the range of values that are scaled to the range [0,255].

Displaying an HDR Image: Tone Mapping

After implementing and experimenting with the the linear mapping you'll realize that this is not a very effective way of displaying HDR images. A non-linear mapping from the HDR values to the LDR values can show detail at more levels. Here we'll describe a very simple tone mapping algorithm which is pretty effective. Your job is to implement it.

This algorithm is based on the key of a scene. The key of a scene indicates whether it is subjectively light, normal, or dark. High-key scenes are light (think of a white room snow), normal-key scenes are average, and low-key scenes are dark (think of night shots). This algorithm uses the log-average luminance as an approximation to the key of the scene. The log average luminance is defined as

Lw_avg = exp( sum_over_i( log(delta + Lw(i)) ) / N )

where i indexes pixels, Lw(i) is the luminance of the pixel (i.e. Ei), and N is the number of pixels. delta is a small value which ensures we don't ever take the log of 0. We then scale all pixels using this value and a user defined key setting, a:

L(i) = (a / Lw_avg) * Lw(i)

So far all we've done is scale the values. We now need to compress all possible values into the range [0,1]. To do this we do the following

C(i) = L(i) / (1 + L(i))

When L(i) is 0 C(i) is also 0. As L(i) gets larger, C(i) gets larger as well, but its upper limit is 1. Finally we can scale this to the range for STPixels and store it in the output image.

Code this algorithm in the tonemap() function.

When you run the program with

hdr -tonemap photo.pfm

it will use this display scheme. Use +/- to increase/decrease the key value.

Hints for Getting Started

Grading

  • 1 stars -
  • 2 stars -
  • 3 stars -
  • 4 stars -

Submission Instructions

We would like submission to be in the form of a single .zip archive. This archive should contain your modified version of hdr.cpp, a text file containing answers to assignment questions, and your photo stack.

Please email this zip file to cs148-win0607-staff@lists.stanford.edu before the deadline. Please make the subject line of this email "CS148 Assignment 5 Handin".

Recent