Assignment 4: Sampling Efficiency

In this assignment you will evaluate the efficiency of different sampling strategies for Monte-Carlo integration. While there is very little coding in this assignment, you will probably want to make scripts to generate the scene files and collect the right data. The data collection itself may take some time, so plan ahead to make sure you have time to complete rendering. You can download the necessary scene files and reference images here.

Step 1: Sampling and Variance

Variance of a random variable $X$ is defined as $E[ (X - E[X])^2 ]$. To compute the variance of an image rendered by pbrt we can compare it, pixel by pixel, to a reference image that we consider to be ground truth for $E[X]$. We've included three EXR images that will serve as ground truth for this assignment (quads_groundtruth.exr, quads_zoom_groundtruth.exr, and frame25_groundtruth.exr). These scenes were rendered with 16384 samples per pixel to ensure that they are very close to the correct image. To measure the variance of any image you render, you can use the imgtool program that you modified in assignment 3:

imgtool diff image_ref.exr yourimage.exr

This will print something like:

    Images differ: 297985 big (98.54%), 301493 small (99.70%)
    avg 1 = 0.0617386, avg2 = 0.0156974 (293.304010% delta)
    MSE = 0.0060754, RMS = 7.794%

Look at the code for imgtool and convince yourself that the MSE value it calculates is equivalent to the variance.

As we increase the number of samples taken, we expect that the variance will down. We will measure this change empirically by modifying the number of samples parameters for the Sampler in the scene files. The file quads.pbrt contains a scene with a single area-light, a square occluder, and a large ground plane:

Render this scene varying the number of samples from 1 to 1024, and compute the variance of each image. You only need to measure square powers of 2 (1,4,16,64,256,1024).

The scene file is setup to use a uniform random sampler. Let's compare this to a stratified random sampler, and a sampler that uses the low-discrepancy Halton sequence, by changing the Sampler and finding the variance for each image.

Deliverables

First, submit a graph showing the variance for the random, stratified and low-discrepancy samplers as you increase the number of sampler from 1 to 1024 for the quads.pbrt scene. You will probably want to use log scale for both axes since you are only considering powers of 2. Furthermore, you should answer the following questions:

  1. When using Monte-Carlo sampling with uniformly sampled random variables, how does the variance change as a function of the number of samples taken, N? Does the data you collected reflect this behavior?

  2. How does the halton sequence sampler perform in comparison to random sampling? Why does distributing the samples evenly through space change the variance in comparison to random sampling?

  3. How does the stratified sampler perform in comparison to the other sampling methods?

Step 2: Stratified Light Sampling

The "directlighting" Integrator, by default, shoots one shadow ray to each light source for every sample per pixel. If you have 10 lights and 32 samples per pixel, this will result in 32 primary visibility rays per pixel and 320 shadow rays per pixel. This is in contrast to the standard path tracing Integrator, which always evaluates one path per sample. But using this we can simulate stratified sampling on just the area light on both quads.pbrt and quads_zoom.pbrt (which is the same scene, just with a different camera) scene. Create new variants of the scenes so that the one square area light is broken up into 16 square area lights arranged in a 4x4 grid covering the same area as the original area light (and using the same power). By rendering lots of samples, these new versions of the scenes should converge to the same image as the old versions. Note that there is no quad primitive; squares are represented as two triangles, and pbrt treats each triangle as two separate lights, so your new scenes will cast 32 shadow rays per pixel sample (and the original scenes will cast 2 shadow rays per pixel sample).

Render the new scene variants with 4 samples/pixel, and the original scene variants with 64 samples/pixel (all using the "random" Sampler). All of these images should use 128 shadow rays per pixel. We'll compare their variances and where the errors appear in both version.

Deliverables

Report the variances for the two variants of each of the two scenes in a (2x2) table. Provide "diff" images of these four scene variants (use the --outfile parameter of the imgtool diff tool to create the diff image with respect to the ground truth Make sure to update pbrt to the latest version from git, only recent versions have fixed saving the diff images). Answer the following question:

  1. Which of the two scenes sees the most drastic improvement when switching to the stratified light sampling stategy? Why?

Step 3: Importance Sampling Efficiency

In lecture, we discussed two improtant concepts that we will explore here: importance sampling and efficiency. Before we begin, take a look at the ground truth image for measure-one/frame_25.pbrt (below), and if you'd like, the original video it was used in: ZERO-DAY.

Previously in class, we demonstrated we could drastically improve convergence of the radiance integral by evaluating the hemispherical reflectance integral at a point by directly sampling an area light source, instead of sampling the entire hemisphere of directions (we, in fact, showed this difference on the quads scene). This is a type of importance sampling: when evaluating direct illumination at a point we know the result will be zero in all directions besides those going to the light source.

This scene contains over 8,000 area lights. If we want to sample light sources instead of the entire hemisphere of directions, we need to choose some probability distribution of which lights to sample for our importance-sampled Monte Carlo estimator (this wasn't an issue in the quad scene with only one light, we trivially chose to sample from the only light source with probability 1).

pbrt has three built-in methods for choosing light sources when evaluating direct illumination (which you can choose by altering the "lightsamplestrategy" in the .pbrt file): "uniform", "power", and "spatial".

"uniform" samples all light sources uniformly, "power" samples light sources according to their emitted power, and "spatial" computes light contributions in regions of the scene and samples from a related distribution. The implementation of "spatial" is somewhat involved, but basically the scene is broken up into voxels (roughly cubic volumes); the amount of light coming into each voxel from each light is sampled (at 128 quasi-random points within the voxel, with no visibility calculations), and a probability distribution over lights, where the probability of sampling a light is proportional to the amount of light contributed to each voxel (with some non-zero minimum), is created for each voxel.

Compute the variance and efficiency for the measure-one/frame_25.pbrt scene and the original quad scene using all three approaches at 512 samples/pixel (use the "random" sampler).

Estimate the number of samples you would require to get the same quality as the "spatial" result using "uniform" and "power".

Deliverables

You should provide the variance and efficiency numbers you calculated for each approach; as well as the

  1. How long would it take to render an equal-to-spatial quality version of the image using the "uniform" strategy? How about using the "power" strategy?

  2. Describe a scene where the "power" strategy would be significantly worse than the "uniform" strategy.

  3. Describe a scene where the "spatial" strategy would be significantly worse than the "power" strategy.

Bonus

See if you can improve the efficiency of the "spatial" light sampling approach by taking into account visibility in some way. There's even a TODO directly in the code about it; see SpatialLightDistribution::ComputeDistribution() in lightdistrib.cpp.

Submit a brief description of your approach, the measured improvement in efficiency on the measure-one/frame_25.pbrt scene, and the code you changed for up to 1 bonus point.

Submission

To submit your work, create a writeup with your plots uploaded and answers to the questions posed above. Use the assignment submission page to upload and submit your work. You can make edits if you wish to make changes after your initial submission; submission time will be taken to be the final edit time.

Grading

This assignment will be graded on a 4 point scale:

  • 1 point: A writeup with major errors.
  • 2 points: A correct writeup but step 1 or step 2 is incomplete.
  • 3 points: A complete writeup with but explanations are not clear, or contain minor errors.
  • 4 points: A correct, clear, and complete writeup

And up to 1 bonus point for the bonus in Step 3.