Marc Levoy
Computer Science Department
Stanford University
Click on a picture to see an uncompressed .tif file.
Click on the .tcl file to edit the spreadsheet source code using Emacs.
Click on the .sht file to run the spreadsheet program on that .tcl file.
Backprojection (without filtering) of a white polygon at the st plane. Verification that -halfopen -cosine -normalize are yielding perfectly seamless and homogeneous fields. Compare to epi_4plane_nocosine_nonormalize.tif.
4-slab orthographic flatland light field of untextured jackInTheBox.iv, which is a slightly modified version of SGI's standard model (in particular, the lighting has been changed), v=127, t=127 (I think), slab resolution = 256 x 256, scene resolution = 512 x 512.
Backprojection (without filtering) of front (c1) and right (c2) slabs, and sum (d1) of the four backprojections. Beautiful, but not very useful. Oversized image best viewed with "drgb" command, which invokes spreadsheet.
Backprojection (without filtering) of alps.tif mapped onto the st plane. Epipolar image is thus a set of perfectly vertical stripes. Two zooms compare a scanline through the original texture with a scanline through the backprojection. They are nearly identical, proving that the backprojection is correctly aligned and resampling well.
epi_4in1point_convolve.tcl
epi_4in1point_convolve.sht
4-slab synthetic flatland light field in which each epipolar image consists of a single-pixel-wide vertical line, thus simulating a perfect view of a point source, slab resolution = 256 x 256, scene resolution = 512 x 512.
Backprojection filter (from Tomographic Imaging, by Kak and Slaney, p. 67, eq. 48, fig. 3.12) is shown at bottom. Left image at top shows backprojection, and right image shows filtered backprojection. Left image shows classic 1/r falloff. Right image is a perfect reconstruction. Voila, a CT reconstruction engine! This version erroneously misaligned the backprojections by 1/2 pixel. See epi_4in1point_convolve2 below for the fixed version. This version also doesn't control absolute output level. See epi_4in1point_convolve3 below for the fixed version. Note: all backprojections done between here
epi_1point_convolve_streaks.tcl
epi_1point_convolve_streaks.sht
1-slab version of above. Only filtered backprojection is shown, which exhibits the streaks that are chararcteristic of backprojection from an incomplete set of projections.
epi_4in1orthoflatjack_convolve.tcl
epi_4in1orthoflatjack_convolve.sht
Same data as 4orthoflatjack. Shows cross section, backprojection, and filtered backprojection. However, filtered backprojection is incorrectly applying the filter to all slabs in a single convolution (see explanation in orig_v_back2).
epi_4in1orthoflatcubes_back.tcl
epi_4in1orthoflatcubes_back.sht
4-slab orthographic flatland light field of cubes.iv with /usr/share/data/textures/patterns/celtic.4.bw texture, v=127, t=127, slab resolution = 128 x 128, scene resolution = 256 x 256.
SceneViewer view of slab 0 geometry is behind at right side. Note that focal plane splits frontmost white cube, and is behind sphere and in front of green and red cubes. This plane will be at pixel y=127 in all reconstructions.
Cell a1 shows the epipolar image for slab 0. The others are "behind". Cell d1 shows filtered backprojection of original epipolar image in a1. Cell f2 shows cross-section of scene, as computed by ivlif, for comparison.
Cells e4, f4, and g4 show range-expanded versions of filtered backprojections (in cells h1, i1, and j1) of grayscale mattes designed to emphasize epipolar features at three angles: right-leaning (+25 degrees), upright, and left-leaning (-25 degrees), respectively. These in turn correspond to scene features at depths that appear at reconstruction scanlines y=166 (deep), y=128 (at the focal plane), and y=91 (shallow), respectively. Note that scene features of the appropriate depth are strong in each.
The mattes were derived by computing the absolute value of the gradient (i.e. the gradient magnitude) of the original epipolar images using rotated 5 x 5 derivative-of-Gaussian filters, one of which is visible along the bottom of the spreadsheet. Then, the maximum of R,G,B gradient magnitudes was taken and thresholded/ramped to suppress low values and emphasize high ones (ramp was {{0 0} {64 0} {255 255 0.8}}). Blurring the maximum before ramping was also tried, but is not shown here. Blurring reduces dropouts, but may introduce false positives into the matte. Throughout this process, each epipolar image, i.e. each slab, is processed separately.
Cell c3 shows the backprojection filter. The resolution of the epipolar slices (=128) (i.e. of the projections) and of the scene (=256) were matched in accordance with the ratio (2:1) of focal plane width to scene width. Note that this reconstruction filter only works if these resolutions are thus matched.
This experiment suggests that the rotated gradient magnitude is a good operator for implementing the angular smoothness constraint. Note that the rotation angle should change for each depth, i.e. for each voxel row in the scene during backprojection. However, given what appears to be a slow variation in response of the operator to rotation angle, it seems reasonable to discretize these rotations, i.e. one rotation and hence one gradient magnitude computation might suffice for processing several voxel rows.
Cell c4 approximately this idea very roughly simply by summing the three backprojections in e4, f4, and g4. A thresholded/ramped version is shown in cell b4. Although these features correspond roughly to certain features of the scene, they are distorted by streaking. If only one slab is processed, these streaks are severe, due of course to the lack of lack of a complete set of projections. In this 4-slab demonstration, they are smaller but still evident. Occlusions are possibly the cause. Or perhaps with sufficient thresholding they are negligable. This reduces the number of non-zero pixels in the matte, but remember that only features corresponding to the three angles included in this experiment should be expected. Cell a4 shows the alignment of the summed backprojections with the cross-sectional image.
epi_4in1orthoflatcubes_orig_v_back.tcl
epi_4in1orthoflatcubes_orig_v_back.sht
A range-expanded version (b6) of the maximum of the R,G,B filtered backprojections of the original (d1) makes a better overall matte than all my machinations with gradients (b4). I hypothesize that this is simply because I have not yet computed gradients at all angles. Note that the features I *have* found are stronger than any due to backprojection of the original, as shown in the accompanying zooms. Press on!
This version erroneously applied the backprojection filter to all slabs simultaneously. See orig_v_back2 below for the fixed version.
(files lost)
Spreadsheet fixed to apply backprojection filter to each slab separately, not to all slabs in a single convolution. Miraculously, the features in the 4 slabs in the gradient experiments were sufficiently disjoint in pixel x and y that they didn't interfere due to this bug. The original would also have been computed wrong, but the displayed image was leftover from a previous, correct computaton.
Spreadsheet also improved to backproject grayscale gradients only, without converting them to rgb, and to reconstruct only the center portion of the scene, which is faster.
(image lost)
epi_4in1orthoflatcubes_gradpair.tcl
epi_4in1orthoflatcubes_gradpair.sht
Two gradients of opposite phases, an approximation of quadrature pairs, a la steerable filters. The even filter is the difference of two Gaussians of different sizes. It is sensitive to valley floors and ridges. Cell j9 shows its result on slab 2 (the back side of the scene). The odd filter is the derivative of Gaussian used above. It is sensitive to valley walls. Cell j8 shows its result on the same slab. Cell j7 shows the max of the two at each pixel. Note that they complement one another. Following the theory of quadrature mirror filters, an amplitude (sqrt of sum of squares) is probably a more appropriate way to combine the two results. See 4in1orthoflatbricks_max_v_sum below.
Cells c4 and b4 show the unramped and ramped backprojected gradients when both filters are thus employed. Compare to epi_4in1orthoflatcubes_orig_v_back.tif. The features found at the angles included in these experiments are stronger here. In particular, there are fewer gaps in the front of the sphere. However, the improvement is not breathtaking. It will be probably be more evident when a continuous range of angles is considered.
epi_4in1orthoflatcubes_256_grad.tcl
epi_4in1orthoflatcubes_256_grad.sht
Striped gradient is now implemented. Conclusions are essentially identical to those reached during the original comparison of backprojection of original (bottom row of images) to backprojection of gradient (top row of images) at three distinct angles (as in orig_v_back) - i.e. backprojection of the original looks better. These experiments show 8 stripes, hence 8 angles. This particular example also shows results on 256 x 256 instead of 128 x 128 epipolar images. Also changed is the size of the gradient filters, 9 x 9 instead of 5 x 5. This makes the found features finer, as you would expect. In theory, for larger filters, more angles, hence more stripes, are necessary. Finally, this version performs no ramping on gradients prior to filtered backprojection, only afterwards. Sorry for all the changes in one image.
epi_4in1orthoflatcubes_thinthick_grad.tcl
epi_4in1orthoflatcubes_thinthick_grad.sht
Comparison of gradients using thinner (bottom row of images) versus thicker (2x width) (top row of images) Gaussians. Thinner is better in all respects. Note that it picks up each texture cycle on the sides of the cubes.
epi_4in1orthoflatbricks_postgrad.tcl
epi_4in1orthoflatbricks_postgrad.sht
If surfaces are diffuse, valleys don't become peaks along their length, so the absolute value operator need not be performed before backprojection. Since this is the only non-linear operator applied prior to backprojection, the entire gradient processing process can be moved to the end of the pipeline. This reorganization vastly improves speed, since the equivalent of my striped rotated gradients is performed on the reconstruction simply by applying horizontal (vertical) gradient filter pairs to the voxel rows (columns) of each slab before the reconstructed slabs are combined.
Unfortunately, reconstructions of the non-regular Celtic texture make it difficult to analyze these effects, so I'm switching to a regular texture, z.brick.white.bw. Cell l1 shows a ramped reconstruction with the gradient operator applied before backprojection ("pregrad"). Cell l8 shows a ramped reconstruction with the operator applied afterwards ("postgrad"). Cell l5 shows a ramped backprojection of the original, with no gradient operator.
The postgrad pipeline looks much better than the pregrad pipeline. Since the surface is diffuse, this should not happen. I hypothesize that when if the rotated gradients are not perfectly aligned with features in the epipolar image, filter responses will be degraded. The only exception is along silhouette edges, where slight misalignments are tolerated. This accounts for the relatively strong performance of this pipeline on corners in the scene relative both to the cubes' walls and relative to the postgrad pipeline.
To verify my pipelines, in an experiment not shown, I removed the absolute value code from both pipelines. With this done, the two reconstructions with gradient operators look about the same up to a scale factor, as they should.
This version, like all other results until 4in1point_convolve2, erroneously misaligned the backprojections by 1/2 pixel. See postgrad2 below for the fixed version.
epi_4in1orthoflatbricks_thickpostgrad.tcl
epi_4in1orthoflatbricks_thickpostgrad.sht
Broader Gaussian. This fills in the walls of the cross-section better, i.e. there are fewer holes due to the space between texture features, but the reconstructed edges get thicker. This occurs because voxel rows (columns) adjacent to the correct edge, i.e. rows (columsn) where the reconstruction is only semi-sharp, respond to these broader filters.
Note: in this experiment, there was a lingering error in the even member of the filter pair of the postgrad pipeline, causing this filter to be a no-op. Nevertheless, the other filter worked and the conclusion is valid. This error was fixed and postgrad re-run. The stored .tif file for that experiment represents the fixed filter.
epi_4in1orthoflatbricks_bkg.tcl
epi_4in1orthoflatbricks_bkg.sht
Enclosed the scene in a box with variously colored walls (tan, cyan, gray, white). New scene is in cubes_bkg.iv. Old scene is in cubes_nobkg.iv.
Top row of blowups show backprojection of original (j10) epipolar image of slab 2 (back side) of scene with no background, max of response of gradient filter pair (g10), and final reconstruction. Bottom row show same images for scene with background.
Final reconstructions look nearly identical, proving that they are independent of absolute scene coloration.
epi_4in1point_convolve2.tcl
epi_4in1point_convolve2.sht
Repeat of 4-slab synthetic flatland light field of single-pixel-wide vertical line (4in1point_convolve), after adjusting backprojection code to insure that central lines in all backprojections cross at precisely the same point in the scene.
This fix applies to even and odd-width epipolar images and scenes, but even-width epipolar images (and scenes) are shifted by 1/2 pixel during backprojection to make the lines cross, so some blur is inevitable. In this case, the "focus of projection" is inbetween two pixels. For odd-width images and scenes, the focus of projection is at the center of the central pixel in both, so no resampling is performed. Thus, filtered backprojection of epipolar images containing a one-pixel-wide stripe result in a 1x1 voxel scene (up to minor numerical error). This is evident in the .tif file.
Note: OpenGL does not support reading and writing of odd-width scanlines, so I had to insert alot of ugly padding code into ivlif to make this work. Nevertheless, I recommend using only odd-width scanlines from now on, to avoid the 1/2 pixel shift and consequent blurring discussed here.
In a separate experiment (not shown), it was also verified that specifying clipping boundaries on the output scene does not adversely affect the now-perfect alignment of backprojection.
This version doesn't control absolute output level. See epi_4in1point_convolve3 below for the fixed version. Also, backprojection of sheared views also doesn't produce correct results, at least as implemented. See epi_4in1disk_convolve2 for a successful backprojection using cylindrical light fields.
epi_4in1orthoflatbricks_postgrad2.tcl
epi_4in1orthoflatbricks_postgrad2.sht
Repeat of postgrad pipeline (postgrad) with improved backprojection alignment. Comparing the two, it is evident that the improved version is sharper. In particular, the scanline (at y=136) corresponding to the back of the green cube in the cross-section (j7) is the sharpest in the filtered backprojection (i10) (and no other scanline is as sharp). Note also that this same scanline is sharpest in the gradient (f10) and the un-ranged-expanded sum of the gradients of all slabs (k8), showing that the convolutions with their cropping are also correctly aligned.
Note that the sum of the gradients is not as sharp as the single gradient. This is because I am taking the sum of the gradients. See maxpostgrad below.
epi_4in1orthoflatbricks_maxpostgrad.tcl
epi_4in1orthoflatbricks_maxpostgrad.sht
In this experiment, I took the maximum of the postgrad gradients of the four slabs instead of the sum. Cells k8 and k9 show the un-expanded and range-expanded sum, and cells l8 and l9 show the un-expanded and range-expanded max.
The max seems more useful. However, it masks the reinforcement in opacity matte that should occur when we see a feature in multiple slabs, in particular the corners of the cubes. Of course, the "sum" version still has maxes, of R,G,B, and of the two gradients in the gradient pair. See max_v_sum below.
epi_4in1orthoflatbricks_max_v_sum.tcl
epi_4in1orthoflatbricks_max_v_sum.sht
This experiment shows the effect of totally avoiding max functions. Cells k8 and l8 show un-expanded and range-expanded postgrads using max of the gradient pair, max of R,G,B, and max of the four slabs. Cells k9 and l9 show un-expanded and range-expanded postgrads using sum of the gradient pair, sum of R,G,B, and sum of the four slabs. Absolute value is still used within each gradient, however. This sum of absolute value is closest of all the experiments I have done to the formal definition of the magnitude of the response of a quadrature mirror pair of filters, i.e. sqrt of sum of squares.
Although sums are preferrable in theory, I have no opinion at this point of which approach is better in practice. Time to start iterating!
See epi_max_v_sum2 for more experiments along these lines.
epi_4in1orthoflatgrid_aliasing.tcl
epi_4in1orthoflatgrid_aliasing.sht
This experiment tries to create as sharp a scene as possible by using very sharp grid textures produced by "slicing" a den program "map" "filled" with "planes". It also compares the effects of rendering the .iv file with and without antialiasing and texture filtering.
The top row uses grid_2_32.bw, a 32 x 32 texture of white dots spaced 2 pixels apart in X and Y on a black background. This row was rendered in ivlif with no antialiasing and texture quality (manually set in cubes.iv) of 0.1. The middle row uses the same texture, rendered with default (4x4) pixel antialiasing and highest texture quality (of 1.0, which includes mipmapping). The bottom row uses grid_2_16.bw, a 16 x 16 version of the same texture. Mapping a half-size texture to the same polygons produces images with texture frequencies half as high. This row was rendered with default pixel antialiasing and highest texture quality. The left column shows the epipolar images for slab 2 (back side), the middle column shows backprojections of the original, and the right column shows range-expanded (with identical expansion code) mattes. Pure summing was used in the gradient pipeline (see max_v_sum above), but a max may have been taken of the 4 slabs - I'm not sure.
The high-frequency texture has a 2-pixel cycle in the epipolar image (i.e. every pixel is different), and it yields a 1-pixel-thick line in the reconstruction. That's good to see. However, note the severe aliasing artifacts elsewhere in the scene. Most interesting! If I remember my CT theory, their location in the scene is probably related to the Fourier transform of the aliasing. Performing antialiasing and texture filtering reduces but does not eliminate these artifacts. After all, we are right at the Nyquist frequency, and the antialiasing being performed is not ideal.
The low-frequency texture has a 4-pixel cycle in the epipolar image, and it yields a 2-3 pixel thick line in the reconstruction. No aliasing artifacts are visible, even in the unramped reconstruction (cell k8, not blown up here). This, then, depicts the resolution limit of my technique.
epi_reproject.tcl
epi_reproject.sht
First attempt at iteration. Looks really cool! 5:15am on 6/7/97. Good night. Wrong, however. Used unfiltered backprojections throughout, and used max of slabs' RGBs instead of gradient to compute new occupancy mask. See epi_reproject2 below.
Cell f1 backprojects using the occupancy volume in cell e1. Without the use of an epipolar erasure mask, the result should simply be an occupancy-masked backprojection of the original. To verify this, I multiplied such a backprojection (d1) by the occupancy volume (e1). The result (shown unexpanded in e3 and expanded in e4) is nearly identical to e1 and e2. The slight differences are probably due to numerical error during backprojection.
Second attempt at erased backprojection. Used filtered backprojection this time. Only one iteration. Looks excellent.
Cell e2 shows ramped filtered backprojection of original slab 0 (front), to show which foreground objects (mainly the sphere) smear over which background objects (all the cubes). Cell i2 shows a ramped occupancy-masked filtered backprojection. (I should have displayed cell h2, but since only one slab was computed in this experiment, the two cells are the same.) Cell n2 shows a ramped occupancy-masked and visibility-masked (i.e. erased) filtered backprojection. Note that the effect caused by the white color of the sphere is removed from the cubes. This removal of color pollution is exactly what we want to see.
Cell k1 shows the epipolar pixel visibility mask, a.k.a. the erasure mask, after the backprojection is complete. Dark pixels here indicate epipolar pixels that are not visibile at the final voxel row (the back of the scene) and thus no longer contribute to the backprojections. Cell l1 shows the corresponding shadow map, a.k.a. light strength buffer. Dark pixels here indicate voxels that few camera views reach.
epi_opacity.tcl
epi_opacity.sht
Here is a table showing the statistics obtained by varying the main free parameter, the scaling applied to the occupancy mask. As expected, raising the scaling (i.e. making the occupancy mask more opaque) makes the epipolar erasure mask and shadow map darker, but they remain nicely antialiased in all the experiments performed.
occupancy %occupied %visible %backprojected time scaling 0.5 7 97 6 28 sec 1.0 15 79 11 28 2.0 33 55 14 28
Cell m2 shows scaling of 0.5, cell m6 shows 1.0, and cell m5 shows 2.0. The snoops at the bottom correspond to these three cells. Low opacity (0.5) leads to thinner walls on the resulting cubes. High opacity (2.0) tends to pass more streaking and aliasing from the occupancy mask into the result. With a less aliased occupancy mask, this might not be so undesireable. A scaling of 1.0 seems best. All of these cells show slab 1 (right side) and are ramped. Cell h2 shows a ramped occupancy-masked but not visibility-masked version for comparison. Cells m1 and h1 show the same comparison for slab 0 (front) and an opacity of 0.5. Note: these results are highly variable. A later experiment (epi_back_forward2) suggests just the opposite: lower opacity is better. Alot depends on the quality of the occupancy mask.
Below are some experiments (not shown in the image) varying the occupancy and visibility thresholds. An opacity scaling of 1.0 was used for all experiments. Slab 0 (front) was used.
occ vis %occupied %visible %backprojected time thresh thresh 0.03 0.03 15 75 11 28 0.01 0.03 48 78 35 30 0.003 0.03 95 78 74 36 0.01 0.01 48 87 41 31
Lowering the occupancy threshold to 0.01 lets in more streaks. Lowering it further to 0.003 makes little visible difference. Lowering the visibility threshold makes little visible difference. 0.03 looks good for general use, and 0.01 looks good for conservative use.
epi_reproject2.tcl
epi_reproject2.sht
Fixed all problems in epi_reproject (above). Very encouraging!
Cell f6 shows an occupancy mask computed from the gradient of a filtered backprojection of the original. Pure maxing was used (see max_v_sum above). This becomes our first guess at the occupancy mask. Cell f4 shows a mask computed from the gradient of an occupancy-masked and visibility-masked filtered backprojection. Note how much cleaner! Cell f7 shows another cycle through the entire pipeline. Walls are now getting thinner. This demonstrates the need for a corrector term, i.e. a forward projection. I reperformed this experiment on September 1 with newer backprojection and gradient operators, but obtained roughly the same results. See also epi_silhouette below. See epi_summary.html for a further discussion of the efficacy of this reprojection pipeline, as well as a comparison to McMillan's shape-from-epipolar-volumes algorithm. See epi_reproject3 (in log #2) for experiments on occupancy and visibility normalization during occupancy-masked backprojection.
epi_forward.tcl
epi_forward.sht
First forward projection to compute a new epipolar image. Looks cool!
Cell a1 through a3 show the original epipolar images for slabs 0 through 2 (front, right, and back sides). Cell a5 shows the cross section. Cell c1 shows a filtered backprojection of the original for slab 0. Cell f4 shows the initial occupancy mask as in epi_reproject2 (above). Cells k1, l1, and n1 show for slab 0 the epipolar pixel visibility mask, corresponding shadow map, and ramped occupancy-masked and visibility-masked (i.e. erased) filtered backprojection, as in epi_erase (above).
Cell m5 through m7 show ramped forward projections of slabs 0 through 2. Features in front generally look the same as the original epipolar images, although somewhat aliased. Features in back are often missing. Notice in particular the loss of the sphere (upper-right-most feature) in m7 relative to a3. The cause of this loss is a thin occupancy function at 10:00 on the sphere in cell f4. This appears to be caused by interference from the green cube. Notice how streaks from the green cube's corners "focus" down on the thin section; this probably caused the gradient operator to fail in this area.
This difference (residual) between the original and forward projected epipolar images is a candidate to be backprojected in order to "improve" the occupancy function. See epi_back_forward for alternative backprojection / forward projection cycles.
This experiment later found to be invalid. It was performing back and forward projections at once, which is invalid. All views must contribute to a voxel row (column) before forward projection is attempted. Subsequent experiments performed the projections separately and are correct.
epi_forward_cross.tcl
epi_forward_cross.sht
Forward projection of gold standard - the Inventor-generated cross section. Cell a5 shows the color cross-section and Reg101 shows an occupancy mask created by thresholding it. This mask has opacities only of 0.0 and 1.0. Cells e1 and f1 show the visibility mask and shadow map resulting from using such an occupancy mask for backprojection. Reg247 shows the result of forward projection using an un-occupancy-masked filtered backprojection of the original (shown in cell d1) as the colored scene and Reg101 as the occupancy mask. Reg260 shows the same on the absolute value of the filtered backprojection.
Since the backprojection in cell d1 is from an incomplete set of projections, it contains artifacts in the form of pixels with values < 0.0. Taking an absolute value before forward projecting reduces the severity of these artifacts, improving the forward projection. After all, our goal is to compute the most plausible scene from the available backprojections. Such a scene must be all-positive. However, even this projection is not perfect; the troughs of the texture are too deep. See, however, warning under epi_max_v_sum2 about taking absolute values before the gradient pipeline.
For comparison, Reg266 shows the result of forward projection using the color cross-section in cell a5 as the colored scene. This result does not contain any backprojection artifacts. The colors don't match the original epipolar image (cell a1), but this is undoubtedly because the lighting of the cross section differs from the lighting of the flatland scene.
See epi_forward_cross2 for an improved version of this experiment with many, many problems solved.
epi_back_forward.tcl
epi_back_forward.sht
This experiment compares two possible initial backprojection / forward projection cycles. As a gold standard, cell m2 was computed using the cross section for both color (cell a5) and occupancy (cell b5) as in epi_forward_cross above.
Cell m3 was computed by forward projecting the absolute value of an unmasked filtered backprojection of the original (cell h1) using an occupancy mask (cell c1) derived from that backprojection using the usual (max-based) gradient operator. Note that this mask was derived only from a backprojection of slab 0, not all four slabs, unlike some of the previous experiments. The equivalent colored scene, i.e. the product of cells h1 and c1, is shown in cell b6. This is what is being forward projected.
Cell m1, on the other hand, was computed by first deriving the mask (cell c1) as above, then backprojecting the original using the mask to produce a masked filtered backprojection (Reg51), then forward projecting the absolute value of this. This method used here is the same as was used in epi_forward except that that mask used there was based on all four slabs and the method used there did not take the absolute value of the backprojection before turning around and forward projecting.
The principal difference between cells m3 and m1 is that the latter is free of color pollution, as first described in epi_erase. In particular, the reds and greens in m1 more saturated.
Not only in this first iteration but in general, it seems appropriate to forward project a colored scene that has had the benefit of the best available masking during backprojection. Although this does involve backprojecting twice before forward projecting, only the second backprojection is masked, so there is no danger of excessive thinning. Moreover, this approach employs the opacity operator during both projections, thus making the new epipolar image a valid replacement for the original and hence a worthy comparison to it.
See epi_back_forward2 for a repeat of this experiment using occupancy masks derived from all four slabs, among other changes and improvements.
epi_4in1point_convolve3.tcl
epi_4in1point_convolve3.sht
Repeat of 4-slab synthetic flatland light field of single-pixel-wide vertical line (4in1point_convolve2), increasing the width of the filter to equal the width of the image and scaling it to make the absolute level of the reconstruction equal the level of the original.
Cell a1 shows the original. The single-pixel-wide vertical line has a value of 255 and it is embedded in an image of width 255 pixels.
Cell b6 shows a 255-pixel wide backprojection filter scaled in value by 255.0. This scaling was empirically determined to produce the correct absolute levels in the reconstruction. That this 255.0 coincides with the width of the filter may be important, but I have no derivation to back up this hypothesis. At the least, it is obvious that increasing the filter width decreases the integral (actually sum) of its response monotonically, and thus the correct value scaling must depend in some way on the filter width. That this 255.0 coincides with the value of the vertical line in cell a1 is irrelevant. I have altered this value, and the reconstruction still works. By the way, this scaling causes the maximum value in cell b6 to be 255.0 and its integral (sum) to be 0.807397.
Cell a4 shows the result of convolving a1 by b6. To make this convolution work without boundary artifacts, a1 was embedded in a larger image padded on the left and right sides by an amount equal to the full width of the filter (255), convolved -nocrop by b6, then the central portion was extracted again.
Cells d1 and d2 show unfiltered and filtered backprojections as before. The absolute level of the central pixel in d2 is exactly 255.0. This is the desired effect.
By the way, increasing the width of the filter from 9*2-1=17 to 128*2-1=255 has no visible effect on the quality of the reconstruction. In a separate experiment (not shown but present in the .sht and .tcl files), filters of
different widths were tried and the resulting reconstructions scaled up in than 1 part in 255, i.e. invisible in an 8-bit image.
Backprojection of sheared views also doesn't produce correct results, at least as implemented, but this isn't evident when backprojecting points, which are essentially delta functions. Nevertheless, see epi_4in1point_convolve4 for a revised backprojection using cylindrical light fields.
epi_4in1disk_convolve.tcl
epi_4in1disk_convolve.sht
Experiments with absolute levels in reconstructions. First, I tried reconstructions of vertical epipolar stripes of various widths.
width of input image reconstruction stripe maxval sum maxval sum 1 255 102 255 781 3 151 309 124 2284 5 140 516 101 3858 7 126 724 92 5392 255 127 44
Input means filtered epipolar image. Input sums are of single slab, single component, so equal to cell a4 / 12 Reconstructions sums are similarly cell d1 / 3. Remember that reconstruction is 511 x 511, whereas input is 255 x 256. The problem with this experiment is that vertical stripes of finite width don't correspond to any plausible scene, as opposed to my one-pixel-wide stripes which I can think of as being delta functions, i.e. points. I fix this next.
Next, I tried reconstructions of epipolar views of disks. Computing sheared X-rays of a disk is tricky. It gets wider at top and bottom (see cell a1). I left the level the same at top and bottom versus the middle, counting on cosine correction during backprojection to make up for this. Here are the results.
radius of input image reconstruction disk maxval sum maxval avgval density (sum/# pixels) 8 36.13 1468 19.472 18.82 0.3467 16 20.76 2986 9.798 9.42 0.6934 32 12.90 6205 5.440 4.66 1.3943
avgval means excluding the spike at the very center of the reconstructed disk. In both experiments, the sum (or density) of the reconstruction is observed to be proportional to the width of the sum of the epipolar image, which for these objects is also proportional to their width in the scene. More importantly, the absolute level (represented here by avgval) is inversely proportional to the width of the objects. If the integrated density dropped as the disks grew bigger, which would be true in reality, avgval would stay the same!
By the way, (linearly) integrated density is the appropriate measure, not (exponentially) attenuated opacity. I tried computing exponential attenuation for each pixel in the epipolar image. This makes the reconstruction dish (not shown). In retrospect, this makes sense; reconstruction is a linear process; it should be performed on (linearly) integrated density, not (exponentially) accumulated opacity.
Although I can now predict absolute level, my reconstruction of disks aren't quite circular (see cell d1). This is undoubtedly because my sheared backprojection geometry causes the backprojection filter to vary in size relative to the scene across the input views. I tried widening the backprojection filter for the bottom and bottom scanlines, but this doesn't work; it produces horrible artifacts in the filtered epipolar image. (Cell a4 shows a filtering without this attempt.) See epi_4in1disk_convolve2 for a successful version using cylindrical light fields.
My newly demonstrated ability to tie integrated density to absolute level in reconstruction cannot be applied to opaque surfaces; they are infinitely thin and thus have no integrated density. Their apparent density arises from the scene sampling grid.
epi_forward_cross2.tcl
epi_forward_cross2.sht
Repeat of epi_forward_cross experiments backprojecting and forward projecting the Inventor-generated cross section, this time with many problems fixed. First, I am now using a vertical stripe texture, which produces deeper troughs in my flatland images (and thus in my epipolar images) and in my cross section image. Second, I am willing to apply an empirical absolute level correction to the cross section to make up for variations in lighting and/or antialiasing. In particular, I have empirically matched it to the epipolar images. Third, I am willing to apply an empirical absolute level correction after backprojection, in keeping with the conclusions of epi_4in1disk_convolve above. Fourth, I have removed all non-linear rampings from the pipeline. Fifth, I have fixed an interpolation bug that ruined all forward projections to slab 2 (back), and a subscripting bug that ruined all forward projections to slabs 1 and 3 (right and left sides). Sixth, I have disabled cosine correction during forward projection. This is the appropriate way to compute epipolar images, since it matches how OpenGL generates sheared views of scenes.
Cells a1 and a3 show the original epipolar images for slabs 0 and 2 (front and back). Cell f5 shows the result of backprojecting all four epipolar images, summing the results, taking the absolute value, and applying an empirical level correction. Before level correction, the maxval was 118. Cell a5 shows the cross section. Before level correction, the maxval was 63. Both were corrected to 255. Cell b5 shows the derived occupancy mask. Cell l5 shows the result of backprojecting cell a5 [cells a1 through a4 (?)] with occupancy mask b5, summing the results, taking the absolute value, and applying an empirical level correction. Before level correction, the maxval was 142. Note that the uncorrected maxval for f5 and l5 are similar, but not identical.
Cells m5 and m7 show forward projection of the level-corrected cross section a5 subject to the occupancy mask b5. Cells m1 and m3 show forward projection of the level-corrected backprojection-derived scene l5 subject to the occupancy mask b5. These four cells have not been level-corrected. Cells m5 and m7 have perfect levels, 255, as they should. Cells m1 and m3 have lower levels, 251 and 167, respectively, reflecting weakneses in the backprojected scene in cell l5. These weaknesses arise naturally from occlusions.
epi_max_v_sum2.tcl
epi_max_v_sum2.sht
More experiments on maxes versus sums in the gradient operator pipeline.
Cell g3 shows sum of the two gradients (in the pseudo-quadrature pair), sum of R,G,B, and sum of the four slabs. Cell g4 shows all maxes. Cell g5 shows max of R,G,B and sums elsewhere. All cells are range-expanded to compensate for their differing absolute levels. The last solution (cell g5) seems best in that it has the highest levels for all cubes and also suppresses the most texture. These conclusions may differ for different scenes. Note that there are absolute value functions within each gradient. These are essential for capturing gradient magnitude.
By the way, do not take the absolute value of filtered backprojections of the original before applying a gradient operator. Since each slab must be processed separately through the gradient pipeline, each backprojection is from a very incomplete set of projections. This implies that there are strong negative-going lobes in the backprojection. Taking the absolute value inverts them, creating a false texture, often of double the frequency of the original and strongly aliased. In another experiment (not shown), I verified that my gradient operators perform poorly on these absoluted-valued backprojections.
See epi_newgrads for even more experiments on gradient operators, including a comparison of twoplane and cylindrical light fields.
epi_back_forward2.tcl
epi_back_forward2.sht
epi_back_forward3.tcl
epi_back_forward3.sht
Repeats of backprojection / forward projection cycle experiment (epi_back_forward). Lots of things have been fixed. These versions also use occupancy masks based on all 4 slabs. I also experiment with scaling the occupancy mask opacity.
In epi_back_forward2, cell g1 shows the occupancy mask scaled to 0.0 to 1.0 and used as opacity, cell g4 shows double opacity (clamped to 1.0, of course), and cell g2 shows half opacity. Cells j7, j6, and j5 show the corresponding occupancy-masked and visibility-masked backprojections of the original. Note that unlike previous experiments (epi_back_forward), the double-opacity mask causes thinning of the cubes' walls in the scene, probably because so little "gets through" the haze of streaks. The half-opacity mask produces the strongest and thus best scene. Cells n1 and n3 (middle row) show forward projections of slabs 0 and 2 (front and back) using the colored scene and occupancy mask to their left, and cells l1 and l3 (top row) show the same for the stronger scene and less opaque mask to their left. The top row looks best, i.e. most like the original epipolar images shown on the bottom row (cells a1 and a3).
epi_back_forward3 continues the series. The bottom row is half opacity, identical to the top row of epi_back_forward2. The middle row is 1/4 opacity, and the top row is 1/8 opacity, which because the opacity masks are temporarily converted to bytes in this spreadsheet is flirting with quantization artifacts (their pixels values are 16 or less). Observe that the in the top rows, features are beginning to shine through. On the other hand, there are no false positives, i.e. no epipolar pixels that should be black (compare to originals at right). This is in keeping with Pat's suggestion to start with low opacity. Correctors for such cases need only add to opacity, never take away.
Time for a corrector term!
epi_corrector.tcl
epi_corrector.sht
First attempt at a correction term.
Cell a3 shows the original epipolar image for slab 2 (back). Cell i8 and d6 show the first occupancy mask and occupancy-masked and visibility-masked backprojection, as usual. Cell i7 shows the consequent forward projection. The difference between cells a3 and i7 was then backprojected (without masking) to yield a "difference backprojection" (not shown), which was in turn added to the original backprojection (which is shown in cell c6) to yield an improved original backprojection (not shown for this iteration, but m6 is the improved original backprojection corresponding to the last iteration). The gradient of this sum was used to compute a new occupancy mask, shown in cell j8. The difference (between cells a3 and i7) was then backprojected again, this time occupancy-masked and visibility-masked (using the new mask), and the result (not shown) was added to the first occupancy-masked and visibility-masked backprojection (cell d6), producing an improved version shown in cell e6. Reg596 shows the consequent forward projection. This entire cycle was then run once more, yielding the improved original backprojection in m6, occupancy mask in q8, the scene in q6, and the forward projection in t3.
The masks are definitely improving, but only subtlely. Note for example the stronger left wall of the rightmost cube and the stronger 10:00 wall of the sphere. A thresholding of the masks is shown below each one. The thresholds were adjusted separately for each iteration so that this left wall barely holds together; for these thresholds, note how much cleaner q8 is than i8 elsewhere in the mask. The scene is improving similarly. In particular, the "difference backprojection" for slab 2 (back) (not shown) was observed to have a particularly strong top wall of of the middle cube, which is weak in the first forward projection (cell i7). This caused both the occupancy mask and scene to be strengthed here, which in turn strengthened this feature in the next forward projection (it is the nearly vertical tan band).
Although iteration definitely improves the forward projections, it has been difficult to distinguish real improvements in the occupancy mask and scene from apparent improvements in the forward projections due to gradually increasing opacity and scene strength. Here are the occupancies and scene strengths:
occupancy mask colored scene First (i8 and d6) 36.7 14.2 Second (j8 and e6) 49.8 16.4 Third (q8 and q6) 74.0 19.5
Opacity and color scaling are deliberately set low (2.0 and 5.0 respectively), allowing the rising opacities and scene strengths to have their effect. I tried deliberately raising these two in the early iterations or lowering them in later iterations to distinguish between the sources of improvement, but the improvements are so subtle that my observations are inconclusive. Cell i3 shows the former case; the opacity and scene strength from the first iteration (cells i8 and d6) were scaled upwards to match the levels of the third iteration (cells q8 and q6 - third row in the table above). The differences between i3 and Reg596 or t3 are pretty subtle.
By the way, the growth of fuzz in the lower-left corner of Reg596 and t3 is worrisome. See epi_corrector2,3 for the next set of corrector experiments.
epi_4in1disk_convolve2.tcl
epi_4in1disk_convolve2.sht
Repeat of attempt to reconstruct a disk from backprojections. This one works!
Cell a1 shows the epipolar image. The radius of the disk is 32 pixels. Note that unlike the sheared views in epi_4in1disk_convolve, these don't need to flare out at the top and bottom. Cell b1 shows the resulting reconstruction. Note that it looks perfectly circular. Note also that there are no streaks at the tangents to the disk! Cell c1 (the label is invisible) below it shows a 10:1 range expansion of the lower part of the range. Still, no streaks.
There is a slight amount of aliasing visible in range-expanded cell c1, but this decreases with increasing numbers of views (verified in an experiment not shown). Note also that the aliasing increases noticeably beyond the circle (barely visible in cell b2) inside which all views contribute. Normalization keeps the level right, but cannot make up for missing data.
Here are some absolute levels for reconstructions of disks of varying radii:
radius of a1 a4 b1 disk maxval maxval maxval 8 255 36.53 23.99 16 255 21.17 11.88 32 255 13.31 5.94
The reconstructed maxval rises linearly with the radius of the disk, which is exactly what one would expect.
This experiment was conducted before ~1/4 pixel offset in backprojections was fixed. epi_4in1point_convolve4 has it correct.
Note added 10/8/97: reconstructions with small filter kernels aren't good. See epi_4in1disk_convolve3 in log #2.
epi_4in1point_convolve4.tcl
epi_4in1point_convolve4.sht
Repeat of 4-slab synthetic flatland light field of single-pixel-wide vertical line (4in1point_convolve3), this time using cylindrical light fields.
Cell a1 shows the original, cell b6 shows a 17-pixel wide backprojection filter, cell a4 shows the filtered original, b1 shows the absolute value of the backprojected original, and b4 shows the absolute value of the filtered backprojection. I didn't expect any surprises here, and I didn't see any. Cell b2 shows a three-camera-view backprojection without absolute value to show that the three views cross correctly, that the backprojection lines are symmetric within themselves and that they follow perfect 45 degree angles as appropriate. (This took some futzing with voxel_offset.) (I use images w/o absolute value because in the absolute value version, the negative lobes near the corners of the scene look like widening of the lines, which they're not.) Cell b3 shows 4 views x 4 slabs to verify that lines from all slabs cross perfectly. Time to start my main experiments again, this time with cylindrical light fields!
epi_newgrads.tcl
epi_newgrads.sht
Even more experiments on maxes versus sums in the gradient operator pipeline, including a comparison of twoplane and cylindrical light fields.
The right half of this image is epi_corrector.tif. Cell i8 from that image is the first occupancy mask. Reg398 replaces twoplane light fields with cylindrical light fields, but changes nothing else. It seems noisier, with circular artifacts. Virtually the entire clipped scene receives contributions from all views, so that's not the cause.
Reg396 shows the effect of changing the gradient operator relative to Reg398. The new operator consists of the sum of the quadrature filter pair, the max of R,G,B, and the sum of horizontal and vertical versions of all these, applied to the sum of the 4 original filtered backprojections from the 4 slabs. This is contrasted with earlier pipelines (and Reg398) in which the horizontal filters were applied only to the front and back slabs and the vertical to the side slabs. The new scheme makes more sense because for example the side views contribute to the definition of the topmost edges of the top cubes, so their contirubtions should be added in before even the horizontal gradient is applied. One obvious difference is that the corners in Reg396 are not as emphasized as they are in Reg398. This is desireable. However, the texture also comes through more strongly, which is not desireable. This issue is investigated more in epi_newgrads2.
Reg624 shows a new scene. The changes are the addition of a tilted cyan cube at the front right (lower right on these images) and a yellow sphere in the back (top), and a finer texture on the white sphere in the front left (lower left). The additional occlusions further weaken for example the inner walls of the red (right) and green (left) cube, and create a major hole in the white sphere at 2:00. Cell g8 (below Reg624) shows the old gradient pipeline applied to this new scene. All these examples are done with an 8.0 x 8.0 x 8.0 511 x 511 x 511 voxel scene and a 4.0 x 4.0 255 x 256 pixel st plane. Neither operator performs well on this scene, undoubtedly due to the greater number of occlusions; both occupancy masks contain many false edges. epi_noise and epi_noise2 explore this issue further.
This experiment was conducted before ~1/4 pixel offset in backprojections was fixed. epi_4in1point_convolve4 has it correct.
epi_noise.tcl
epi_noise.sht
epi_noise2.tcl
epi_noise2.sht
Quick experiments to determine if aliasing artifacts in reconstructions are eliminated by using an irregular texture. They are, but my gradient operators don't work well on noise.
In epi_noise, cell d3 shows a regular texture, c5 shows the backprojection, and cell g7 shows the occupancy mask computed using the new gradient pipeline (i.e. running on the sum of the slabs' backprojections). The bottom row (cells a3, e5, and g8) shows the same sequence where the scene is textured using the noise texture shown in g4. In epi_noise2, the new scene with additional objects is shown. Again, the top row employs a a regular texture, and the bottom row employs a noise texture. (The cell names in epi_noise2 differ from in epi_noise.)
In both experiments, the gradient operator performs relatively poorly on the noise texture. Nevertheless, it is easy to see that certain regular patterns in free space are removed by using an irregular texture, in particular the shadow edges above and below the top edge of the middle white cube and the false diagonal edges well above the top edge of the red cube (top right). These effects are more dramatic in epi_noise2, undoubtedly because there is more occlusion, hence more streaking, hence more false edges in the occupancy mask.
By the way, the streaking when using the noise texture (cell c5 in epi_noise2) appears worse than when using the regular texture (cell e5). This is probably because only the corners give relatively strong readings when using the noise texture, and these cells are automatically range-expanded images, hence the streaking appears brighter.
I also tried improving the response of my gradient operators on noise textures by taking the max of the responses with operators at two scales. Cell h8 shows the response with filters of the standard (sharp) scale, cell g8 shows the response with broader filters (one is visible in cell a6), and cell i8 shows the max of these. I see little if any improvement. The snoop at right is of cell h8.
epi_newgrads2.tcl
epi_newgrads2.sht
More comparison of the new and old gradient pipeline (see epi_newgrads). The new one is definitely better.
Cells c1 through c4 show the new scene, regular texture, cylindrical light field. Cell c5 shows the sum of the four slabs. Cell g8 shows the new gradient pipeline, which runs on the sum in cell c5. This image looks similar to Reg624 in epi_newgrads. Cell q8 shows the old pipeline, which runs on the individual slabs in cells c1 through c4.
As the snoops show, the new pipeline shows the textures along the edges of the cubes, which is undesireable. However, the old pipeline is very prone to responding to the streaks present in the individual slabs, which are relatively reduced in the sum of the slabs used in the new pipeline. This is particularly evident on the 60-degree edge at the lower-right corner of the spun cyan cube (lower-right corner of scene).
Evidently, for scenes containing many occlusions, I should use the new pipeline. See epi_summary.html for a discussion of the relationship between these operators and shape-from-focus operators and other computer vision algorithms.
epi_corrector2.tcl
epi_corrector2.sht
epi_corrector3.tcl
epi_corrector3.sht
Second attempt at a corrector term, with cylindrical backprojection, the new gradient pipeline, and slightly more rational absolute level control.
In general, the correction cycle works as before. The top row shows a sequence of epipolar images. Cell a1 is the original; the others are successive forward projections with relatively low opacity and color scaling (1.0 and 3.0 respectively). The same opacity and color scaling are applied during backward and forward projection, and the same are applied on all iterations. In addition, the color is scaled down so that its maxval matches the maxval of the opacity. The reason for this is that backprojection uses relatively low opacities, yielding very dark colors. The indicated scaling essentially restores the colors to full brightness (rescaling maxval up to 255), then premultiplies them by the maximum opacity, creating the opacity-premultiplied colors expected by the forward projection code. A global scaling suffices here, rather than a voxel-by-voxel scaling, because the colors have already been scaled locally by voxel opacity during backprojection. However, see note at end of experiment about the validity of global scaling.
The next row shows the sequence of difference-enhanced unmasked backprojections, as before. The third row shows the sequence of occupancy masks. The fourth row shows the sequence of difference-enhanced, occupancy-masked, and visibility-masked backprojections, as before.
Again, the forward projections and masks are improving. Note in particular the top edge of the middle white cube, which corresponds to the central, nearly vertical white stripe in the epipolar images. It is weak in the first mask and forward projection, but stronger in the next iteration. Note, however, that the masks also get noisier, leading to fuzz in the forward projections.
In epi_correction3, the occupancy masks underwent non-linear ramping before being used for back and forward projection. In particular, a gradually increasing floor was applied (e.g. 12, 20, 28) and a gradually increasing gamma (1.0, 1.1, 1.2). This keeps the noise low, which results in cleaner forward projections. Note also that weak edges in the mask become stronger. Cell ee1 is an alternative final forward projection (alternative to Reg1835, that is) using a stronger opacity scaling (3.0). It looks pretty good relative to the gold standard in cell a1. Cell gg1 is the result of another set of ramping parameters, but I'm not sure what they were. It somehow seems less aliased than Reg1835, although it contains a bit more fuzz. See epi_ramp_mask for methodical experiments with mask ramping parameters.
Essentially, this algorithm is selectively scaling the colors in the epipolar images such that the gradient of their backprojection yields a mask with stronger walls were they would otherwise be relative weak. However, the weak walls become no better coalesced geometrically. To improve the geometric nature of the masks, I tried blurring them in addition to ramping them. This made no visible difference in the resulting forward projections. However, see epi_color_correct2 for a more sensitive experiment in which it was found that blurring is useful.
The displayed forward projections were the absolute value of the full-range projection. This is invalid; clamping at zero should have been used. This is fixed in subsequent experiments.
It was also subsequently decided that global scaling is invalid. The local scaling can only be treated as valid premultiplication if no global scaling is performed. Otherwise, it does not constitute a valid compositing result. See epi_color_correct5 for a corrected pipeline.
epi_ramp_mask.tcl
epi_ramp_mask.sht
Various rampings of occupancy mask. Only the first forward projection is shown. The scene has been changed slightly. The big white sphere is now untextured and the red bue is untextured, dark, and shiny. Also, the occupancy mask is scaled up to full opacity. Finally, the forward projections shown are clamped at zero. Previous experiments took the absolute value, which tended to double the texture frequencies and was wrong. The epipolar images for this new scene have been strung together to form a 360-degree sinugram, which is shown in epi_sinugram.tif. An Inventor rendering of the scene itself is shown in epi_sceneviewer.tif. The flatland light field plane is displayed using screen-door transparency, with a part of the red camera line corresponding to slab 0 (front slab) visible in the foreground. This line should be a circular arc around the object, but I never updated this part of ivlif when I moved to cylindrical light fields.
Here is a table showing the parameters used. Cell d7 shows the occupancy-masked and visibility-mask color scene for the last case (floor=16, gamma=0.3). Cell a3 shows the original slab 2 (back) epipolar image for comparison.
position forward occupancy floor ceiling power on image projection mask (->0) (->255) (gamma) upper-left Reg785 Reg734 8 192 1.5 top-middle Reg729 Reg678 8 192 1.0 upper-right Reg673 Reg602 8 192 0.5 lower-left Reg1055 Reg1004 14 192 0.5 lower-right i3 g7 16 192 0.3
As expected, high gammas lead to only the strongest surfaces being opaque, and low gammas tend to bring out the weak surfaces, but they also enhance the background fuzz. Very low gammas also darken features somewhat. This is probably because even though the opacity-premultiplied color scene makes no contribution, visibility decreases with occupancy mask opacity, so the loss of antialiasing in the mask means that there is less visibility "available" by the time the rays reach the voxels that really contain color, i.e. the surfaces start to self-shadow.
Raising the floor knocks out the fuzz. In another experiment (not shown), it was discovered that raising the floor too much has the same effect as high gammas - only the strongest surfaces survive, as expected.
In all these results, the texture lines thin out somewhat. This is wreaking havoc on any iteration...
epi_color_correct.tcl
epi_color_correct.sht
epi_color_correct2.tcl
epi_color_correct2.sht
An attempt to get the colors in subsequent forward projections to match the original. Used a textured scene to generate the occupancy mask in the usual way, then used a completely untextured scene (see cells a1 and a3) in the subsequent occupancy-masked backprojections and forward projections. This use of untextured scenes uncovers many interesting subtle problems with the pipeline that I overlooked when using strongly textured scenes.
Cell g7 shows the occupancy mask generated in the usual way. Cell g6 shows a blurred version of this. Without this blurring, the subsequent forward projections are noisy due to the introduction of dark stripes (self-shadowing) where the occupancy mask varies in density. Cell d6 shows the result of occupancy-masked and visibility-masked backprojection. Note the purple artifacts at the corners. Purple is the complement of green. This represents un-filled-in negative lobes due to having an incomplete set of projections. This occurred with the textured cubes as well, but it is more obvious here. Cell d6 shows a zero-clamped version of this. Zero clamping is more correct than taking the absolute value. Cell g5 is cell g6 multiplied by a 0/1 mask derived from cell d6. This zeros the occupancy of any voxels whose color is black. This avoids dark stripes (self-shadowing) in the forward reconstructions where there were negative lobes in the zero-clamped color scene. Cells i1 and i3 show the resulting forward projections.
Cell n5 shows the occupancy-masked backprojected differences, in the usual way, and cell o6 shows them scaled up by an arbitrary amount, added to cell d6, and zero-clamped. Since there is no difference of textured epipolar images on which to run a gradient operator, the original occupancy mask is used. This upward scaling seems necessary because the backprojected difference (cell n5) is too weak to fix the color scene, despite ostensibly representing the required difference. By scaling it up (by 10.0 in this experiment), it corrects the colors well. In particular, look at scanline Y=30 (the bottom corner of the cyan cube). It has coloration artifacts in d6, probably due to incomplete projections. By adding a scaled up color difference, these artifacts are fixed in o6. Without the scaling, they aren't fixed. Cells r1 and r3 show the second forward projection.
epi_color_correct2 continues this process to a third iteration. The sequence of differences (before scaling) and zero-clamped scenes is shown along the bottom, except that cell d5 is not a difference; it is the first occupancy-masked and visibility-masked backprojection. The scaling applied to the difference the second time (going from y5 to z6) is only 2.0. Using 10.0 overcorrected. Each iteration brings us closer to the original (note also that the color scene is improving), but convergence is hampered by flaws in the the original (and only) occupancy mask. Note how the second difference (cell y5) is "desparately" trying to fix problems at the corners of the green cube, problems that cannot be fixed because the occupancy grid is flawed.
This is my first successful iteration. I've learned several things:
Now, I need to improve the occupancy mask geometrically...
epi_badgrads.tcl
epi_badgrads.sht
Tried generating masks from various combinations of rotating gradients, extruded versus non-extruded gradients, and untextured versus textures scenes.
Cells g6 and g7 show non-extruded and extruded rotating gradients (8 angles) on the untextured scene. Cells g9 and g10 show the same gradients on the (partially) textured scene. Extruded looks blurrier and less selective to the real object edges. Cell c9 shows one attempt to combine g6 and g9 by blurring then ramping g9 severely to create nearly a binary mask, then multiplying g6 by this. The hope was to combine the thinness and lack of texture in the edges of g6 with the selectivity for objects (as opposed to streaks) of g9. It works, more or less.
However, none of these results look particularly good. Rotating gradients require large filters, which produce fat edges in the occupancy mask. In experiments not shown here, I found that fat edges in masks cause alot of self-shadowing, leading to dark, poor-quality color scenes. In another experiment (not shown), I tried computing central differences by subtracting original backprojection from a blurred copy of itself. This also produces fatter masks, and it is sensitive to salt-and-pepper noise, as expected.
In general, I don't see much difference between masks generated using a rotating gradient with many angles and masks generated using only two angles, which is equivalent to using only a horizontal and a vertical gradient with bigger filters. Thus, I should probably discard rotating gradients. They are slow anyway. If I want to compute only horizontal and vertical gradients, the filters can be any size I want.
Using a rotating gradient (and large but non-extruded filters), untextured scenes produce nice thin edges in the mask, but strong streaks at changes in visibility. This is a nice visualization of the visibility complex, but it is not a good occupancy mask. Untextured scenes also produce strong corners. Actually, these are normal corners; the textured scenes have unnaturally dark corners because the textures I'm using are black along the t=0 edge. By the way, applying small horizontal and vertical filters to the untextured scene (results not shown) picks up only the corners, not the edges or the streaks. This is useless.
I'm at wits end on these masks. I cannot design a filter for untextured scenes that picks up objects without inter-object streaks, and I cannot design a filter for textured scenes that picks up poorly backprojected areas ("venetian blinds") without screwing up the rest of the mask.
epi_nonfilt_back.tcl
epi_nonfilt_back.sht
Backprojection from incomplete projections is introducing aliasing into my colored cross sections, and from there into my forward projections, residuals, etc. In this set of experiments, I tried using unfiltered backprojection in various places in the pipeline. The scene is a single, axis-aligned, off-center cube.
The image shows occupancy-masked backprojection of unfiltered epipolar images. The mask is the usual ramped blurred horizontal and vertical gradient of a filtered backprojection of a textured scene (cell g5). This pipeline yields a blurry colored cross section (cell d5), which in turn blurs the forward projection (cell g1).
I also tried the following experiments (not shown):
None of these pipelines reduce the presence of a texture pattern in the mask. If the scene being backprojected with the mask is untextured, then the pattern in the mask creates texture in the cross section and more (and different) texture during forward projection (due to self-shadowing). If the scene being backprojected with the mask is textured, then the pattern in the mask combines with it in the cross section, creating either a doubly deep texture (if they align) or aliasing (if they don't), as well more and different texture in the forward projection. I need a smoother mask. Perhaps I need a polygonal mask.
These experiments suggest that two factors are pivotal to my failure to achieve a convergent pipeline:
See epi_color_correct3,3a,4,4a below for a possible way to alleviate (1).
epi_color_correct3.tcl
epi_color_correct3.sht
Attempt to use iteration on unflitered backprojection instead of on filtered backprojection to generate a forward projection that matches the original, to avoid the streaks and color artifacts associated with filtered backprojection from incomplete projections. The scene is the same single, axis-aligned, off-center cube used in epi_nonfilt_back.
Before performing this experiment, I decided (incorrectly) that the color cross section should be multiplied by opacity during forward projection. I reasoned that it cannot be assumed that it is perfectly premultiplied. As it turns out, this is true, but the cross section *is* nevertheless premultiplied. In epi_color_correct5, I fix the entire premultiplication problem.
In epi_color_correct3, cell j9 shows a filtered backprojection and cell g5 shows the usual mask. Cell d6 shows the colored cross section resulting from masked unfiltered backprojection, and cells i1 and i3 show the front and back slabs of the first forward projection. A correction step consists of computing a difference cross section via unfiltered backprojection of the difference between the forward projection and the original epipolar image, then adding this difference cross section to the most recent cross section to obtain a new cross section. Cells q6, bb6, etc. and t1, t3, ee1, ee3, etc. show subsequent iterations. The iteration suceeds in eliminating the blur inherent in unfiltered backprojection, i.e. it is a valid replacement for filtered backprojection. This is to be expected, according to the literature on ART.
In epi_color_correct3a, the bottom row (numbered registers) is the same as the bottom row in correct3. The top row show the effect of starting from a colored cross section obtained using filtered instead of unfiltered backprojection. This colored cross section is shown in cell d6. This iteration also converges, although slower.
In this single-cube example, there are unfortunately no visible color artifacts in cell d6. However, in another experiment (not shown), I used filtered backprojection in all iterations. In this case, all iterations exhibited the artifacts evident in the poor forward projection in cell i3. In fact, they all looked the same - there was no convergence occurring at all! However, see below for a conflicting result.
In epi_color_correct4 and 4a, I repeated 3 and the use of filtered backprojections in all iterations (the experiment not shown above) for the usual multi-object scene. In this case, there is convergence in both cases. It is slower if filtered backprojection is used, but the final step is better (the texture is sharper). Hmmm.
By the way, the cross sections in 4a (bottom row) are getting blurrer. In another experiment (not shown), I tried using an unblurred mask (usually cell g7) as my occupancy mask instead of a blurred mask (usually cell g5). This solves this problem. In epi_color_correct4, by the way, they start blurry because unfiltered backprojections are used, so if they get blurrier if the blurred mask is used, I can't tell.
I take this sequence of experiments as proof that iteration to correct the artifacts introduced by filtered backprojection from incomplete projections can be successfully performed using unfiltered backprojection, and it might actually be preferrable to do so. One option not tried is to use unfiltered backprojection in early cycles, followed by one final cycle of filtered backprojection.
This still doesn't help me improve the mask, though.
epi_color_correct5a.tcl
epi_color_correct5a.sht
A major problem in the entire pipeline is fixed. I have been adding the masked colored cross section for the 4 slabs together. This is invalid. They are premultiplied by opacity (during backprojection). Adding them together creates a composite cross section that is too bright, invaliding the compositing during forward projection. I had noticed that pixels in my forward projections were occasionally > 255, but I couldn't figure out why. This is why. To add them together, they must be un-premultiplied, i.e. divided, by opacity. (Or never multiplied by opacity in the first place; see epi_reproject3 in log #2.) Alternatively, it seems justifiable to simply take the max of the 4 slabs. This somehow seems especially justifiable if unfiltered backprojection is used, in which case there are no negative values in the individual slabs' cross sections. In keeping with this cleaner pipeline, I am removing from the Tcl code registers any provision for global scaling of the colored cross section just prior to forward projection. This experiment shows the results.
In epi_color_correct5, the top and bottom rows show iteration (using unfiltered backprojection of differences, in keeping with the previous set of experiments) of the untextured and textured scene, respectively. max is used instead of adding to combine colored cross sections. The unblurred occupancy mask is used. These results look great. The pipeline really converges, to the extent permitted by the flawed mask.
In epi_color_correct5a, filtered backprojection is used. It converges slowly, mostly because it needs to increase in brightness. But scaling the colored cross section prior to forward projection is dangerous; it violates the compositing algebra. As proof of the need to increase brightness, cell i3 represents a first forward projection with the colored cross section scaled upwards by 6.0 before forward projection. It looks like a much later iteration. It is also slightly sharper, undoubtedly because multiple iterations introduce slight blurring due to resampling.
My conclusion is that, with the repaired pipeline, either unfiltered or filtered backprojection can be used. If filtered backprojection is used and the colored cross section is brightened before forward projection, which is somewhat dangerous, then convergence is nearly immediate.
This still doesn't help me improve the mask, though.
epi_corrector5.tcl
epi_corrector5.sht
Two fresh attempts to improve the mask. The second one is as good as I think I'm ever going to get unless I apply more aggressive scene hypotheses.
New colored cross sections are computed as in epi_color_correct5 using unfiltered backprojection (cells d5, q5, and bb5 in corrector4), then the ratio of the new cross section to the old cross section is computed. A blurred version of this ratio is shown in cells s5 and dd5 in corrector5. In corrector4, the old mask is multiplied by this ratio (I can't recall if I used the blurred or unblurred ratio), yielding cells f7, q7, and bb7. In corrector5, the filtered but unmasked backprojection of each original epipolar image is multiplied by this ratio, as shown in cells u9 and ff9, and a new mask is computed from scratch, yielding cells w10 and hh10. Some subtle mask renormalizations are performed to keep the mask from fading out. In corrector5, these might not be nececessary. In both images, the top row shows the sequence of forward projections. The first one is fuzzy because unfiltered backprojection is being used. As I have discovered (see epi_color_correct3), later iterations fix this, a la ART. In corrector5, I have also fudged the arbitrary (and dangerous) scaling of the cross section to keep the forward projections from fading.
In corrector4, there is no regularization (smoothing), so the mask becomes more textured and eventually begins to disintegrate. In corrector5, the unmasked backprojections, and hence the mask, are strengthened where they were weak, but the sequence of masks becomes neither more nor less textured. This works well. Note for example that the top of the green cube stays roughly constant while the top of the middle white cube strengthens. In another experiment (not shown), I applied this pipeline to cyl_nobkg, in which all objects are bigger, diffuse, and textured. The improvement is even more evident. The parts of the mask that are supposed to stay constant get a bit chewier, but I suspect that more blurring of the ratio will fix this.
Taking the long view, this result is not much more convincing than epi_corrector, my first attempt at a correction term. In both cases, I am merely scaling different parts of the mask; I am not changing its geometric character.
epi_vector_field.tcl
epi_vector_field.sht
Line integral convolution of a light vector field. Wow!
Cells f7 and d5 show the usual occupancy mask and colored cross section for the standard advanced scene with mostly textured objects. During the masked backprojection to compute cell d5, a light vector field was also computed. The dx and dy components of this vector field for each slab are visible in columns k and l, respectively, of the spreadsheet. These are unnormalized components.
Cell f5 shows the sum of the visibility images. This is scalar light field strength. Dividing the vector dx and dy images by this strength gives columns h and i. Alternatively, cells k9 and l9 contain the sum of the unnormalized components for all slabs, and cells k10 and l10 contain the normalized components. Note: do not confuse the scalar strength with the magnitude of the vectors. A field might be strong (high visibility) but have dx=dy=0.0, if it is equally visible from all directions. A future experiment should try normalizing by the vector magnitudes. (See epi_vec_magnitude.)
Cell k11 and the center blowup show a LIC visualization of the normalized vector field images (cells k10 and l10), postmultiplied by the strength (again) to aid comprehension. Cool image! The boundary regions look wrong; they should be isoangular, not perpendicular to the (square) boundaries of the image. The locations of the sinks are unreliable; vector magnitudes are very small inside the objects. The component magnitudes in cells k10 and l10 vary from -1.0 to 1.0. Best results for the LIC visualization are obtained by multiplying these components by 50 before using them as inputs to LIC. As I have implemented LIC, this scaling creates streaks of length 50.
Cell k12 shows my first attempt to apply a derivative-of-Gaussian (odd) filter (visible in cell h12) to the blurred occupancy mask (not shown) along the field lines (from cells k10 and l10). It generates a decent zero-crossing situation in some places (the green cube in upper-left), but artifacts in others (the cyan cube in lower-right). These artifacts arise because the vector field disintegrates at the cube edges. This problem is addressed and solved below in epi_vec_crossings.
See epi_vector_field2 for an improved version with better normalization.
epi_vec_crossings.tcl
epi_vec_crossings.sht
Experiments with various ways to produce the zero-crossings image required for input to Marching Cubes. The scene is the single, axis-aligned, off-center cube.
Cell f7 shows the usual occupancy mask. Cells k9 and l9 show the *un*normalized light vector field, as opposed to the normalized vector field using to generate the LIC image in epi_vector_field. The resulting vector field (not postmultiplied) is shown in cell l11. Consider the sweep from bottom to top. Note the abrupt change in the vector field at the y=12 scanline. This explains the artifacts in the zero-crossing image in cell l12, generated (as explained in epi_vector_field) by applying a derivative-of-Gaussian filter (cell h12) to the blurred occupancy mask (cell e7).
As an improvement, I tried severely blurring (using a 10 x 10 pixel average) the vector field components k9 and l9 to produce cells h9 and i9, then repeating the process. Since this was applied to the unnormalized components, the effect is to extend the reasonable flow lines outside the cube, which are large, a small distance into the cube, where the components are small. The LIC image (cell k11) shows that the flow lines are reasonable for 3 more scanlines. The resulting zero-crossing image (cell k12) is much better.
By the way, one cannot simply apply horizontal and vertical derivative-of-Gaussian filters to the blurred occupancy mask and sum them. The resulting zero-crossings are not consistently placed with respect to the mask edges, as shown in cell c6. The light vector field is essential to the generation of correct zero crossings!
epi_vec_based_grad.tcl
epi_vec_based_grad.sht
A quick experiment to see if I can generate an occupancy mask using LIC.
Cell j9 contains an unmasked backprojecton of the original single cube. Cells c8 and d8 contain the two phases of the usual horizontal gradient, and cells c10 and d10 contain the two phases of the vertical gradient. Cell f7 contains the usual ramped occupancy mask, and cell e7 contains the usual blurred version.
Cell j11 contains a B&W version of the unmasked backprojection in cell j9. I then applied the same two filters (visible in cells h12 and g11) using LIC. The blurred vector field (cells h9 and i9) was used. Absolute values were taken to produce cells j13 and j14. The sum is in cell h14 and the blurred version in cell g14. This mask compares favorably to cell e7. However, it did not require separate horizontal and vertical convolutions; the vector field steered the filters. In this case, the filters were applied perpendicular to the field lines rather than parallel as in the zero-crossings experiment.
The resulting mask differs from the usual for several reasons. First, the vector field is not perfectly perpendicular to the cube faces at their interface. Thus, the filters are slightly tilted. Secondly, the filters employed were long (as seen in cells h12 and g11) and were applied at many (linearly interpolated) places between pixels in the backprojection. Third, an attempt was made to scale the vector field components such that the scaled length of these filters was 6.0, the same length as the filters employed in generating the usual mask, but this scaling is necessarily approximate. By the way, because of the long filters, this method is actually more expensive than the usual method, but this is not inherent; shorter filters could have been employed.
Note of course that this technique for generating a mask cannot be used until the vector field is available, which in turn requires a previous mask. But it's an interesting experiment anyway. Might prove useful at some point.
epi_vec_magnitude.tcl
epi_vec_magnitude.sht
Normalization of the vector field using its magnitude rather than the scalar light field strength. The usual textured scene is used.
The old pipeline is shown at left. Cells k9 and l9 show the unnormalized vector field components, cell f5 show the light field strength, cells k10 and l10 show the normalized components, cell l11 show a LIC visualization of the field, postmultiplied by cell f5, and the Regxx image below it shows the resulting gradient of the occupancy mask. It is unusable because no blurring of the vector field has been applied.
The new pipeline is shown in the middle for an unblurred vector field and at right for a blurred vector field. Cell g4 (and g6) show the sqrt of the sum of the squared components. Cell h9 and i9 show the normalized vector field, cell k11 shows the LIC visualization, postmultiplied by k11, and cell l12 shows the gradient of the occupancy mask. Cell k12 (the blurred version) is usable, as in vec_crossings, but is more theoretically justifiable.
See epi_vector_field2 below for more images from this pipeline.
epi_vector_field2.tcl
epi_vector_field2.sht
Summary image of the pipeline running on the standard advanced scene.
Cell a1: original epipolar image, slab 0 (front) Cell j9: filtered backprojection of original, sum of all slabs Cell d6: masked filtered backprojection using mask in f7 Cell f7: occupancy mask derived from j9 in the usual way Cell f1: shadow image from masked backprojection of slab 0 (front) Cell e1: visibility image from same backprojection Cell k5: unnormalized X-component of vector field from same Cell l5: unnormalized Y-component Cell f5: sum of shadow images Cell g5: magnitude of vector field (sqrt of X^2 + Y^2) Cell k10: normalized X-component (normalized by magnitude) Cell l10: normalized Y-component Cell k11: LIC visualization of normalized (unblurred) vector field Cell l10: gradient of occupancy mask along (blurred) vector field lines
Time to extract isosurfaces!
epi_contour.tcl
epi_contour.sht
First semi-successful attempt to extract contour surfaces (after many failed attempts). Used original, simpler scene.
Isosurfaces of the zero-crossing gradient images I have been computing don't work. The background level of the image is also zero, so many surfaces are extracted. I can't figure out a way to mask the operation so that only the surface I want is extracted. At the least, in any of these schemes, the inner and/or outer hulls of the occupancy mask lines would also be extracted.
Isosurfaces of shadow maps or vector magnitude maps can in theory be thresholded to yield the correct, single surface. However, the central portion of these maps, especially the shadow map, is dark due to reduced visibility in the center. There is no contour level that extracts the scene without also extracting garbage in the center. Thus, these don't work either.
I finally tried creating a new sort of shadow map by backprojecting along the flow field using an opacity-style visibility calculation. I've thought of this before. Since these flow lines go directly from the boundaries of the scene into one or another wall of the cubes, there should be no excessive darkening in the center of the scene.
I created this "flowed backprojection" by running a multiplicative LIC with a slightly ramped (more than usual, that is) blurred occupancy mask (in cell a2) as the input image (the unblurred mask produced results with alot of aliasing), the blurred flow field (shown in cell d3 postmultiplied by the unblurred vector field magnitude), and a step-function filter (visible in cell b4): 1.0 (i.e. full weight) in its negative half, which is then applied to all voxels "before" the current voxel (i.e. back toward the cameras), and 0.0 (i.e. no weight) in its positive half, which is then applied to all voxels "after" the current voxel (i.e. further along the flow lines, including into the cubes. The result is shown in cell c4, and in reverse-video in cell e4 and the blowup. The resulting contour lines (at level=72) from vtk is shown to the right.
This pipeline works, more or less. The cubes stand out strongly and sharply from a black background (in cell e4) with edges placed where they belong. Weak edges, like the left wall of the cubes in the blowup, are strong enough to make a decent contour. Streaks, visible to the left of the cube in the blowup, don't affect the contours. The only artifact, a serious one, is that where the vector field disappears, i.e. in sinks (in the centers of the cubes) and at saddles (along the diagonal paths between the corners of nearby cubes), the flowed backprojection becomes unstable, producing garbage. This garbage is evident in the contours. See epi_contour2 for a way to fix this.
epi_contour2.tcl
epi_contour2.sht
Theorem: flows from < 180 degrees of camera arc contain no saddles, which were messing up epi_contour. Thus, in this experiment, I tried flowed backprojection from individual slabs, which are each 90 degree arcs.
Cells p5 through p8 show postmultiplied LIC visualizations of blurred vector fields associated with the usual masked backprojection of each slab Note the absence of saddles. Note also that the top-right cube is hollow in cell p8 due to its weak left wall.
Cells q5 and q8 show (for slabs 0 and 3 (front and left side)) flowed backprojections (multiplicative LICs) as described in epi_contour. Each slab has clean centers, but garbage in the shadows. Cell q9 shows the sum of the flowed backprojections for the 4 slabs. This clearly doesn't work. Cell q10 shows the max. This works, but the top-right cube is hollow.
Cells r5 and r8 show the flowed backprojections multiplied by the unblurred vector field magnitude for the corresponding slab, and cell r9 shows the max of this for the 4 slabs. This has no hollow top-right cube, but does contain garbage in the center, although not as much as in epi_contour.
The two reverse-video masks are visible in cells q11 and r10 (r10 has been gamma-ramped by 5.0 in an attempt to suppress the garbage in the center). The two resulting geometries are shown to the left and right, with thresholds of 80 and 56, respectively.
Both of these pipelines work reasonably well. However, the unnormalized pipeline (on the left) is clearly more robust, with a very clean center. The problem here is that the same "max of 4 slabs" operation that eliminates shadows that arise during backprojection of individual slabs also eliminates any space-carving in individual slabs, like that which closes off the top-right cube. Thus, the left pipeline is robust but conservative; it assumes the largest possible unoccupied space. I can't think of a way to differentiate between the shadows and the center of the top-right cube. Recall, however, that previous experiments show that I can improve the occupancy mask somewhat through iteration, strengthening the weak left wall of this cube.
Time to smooth the geometry (vtkSmoothPoly) and try generating a new light field as part of an iteration. Question: how to map texture onto this geometry? Answer: using masked backprojection of the original epipolar images with a scan-conversion of this geometry as the occupancy mask!
epi_contour3.tcl
epi_contour3.sht
Strange but rather successful experiment. If maxes of backprojections from 90 degree arcs have fewer shadows, then how about maxes from the backprojection due to each camera view?! I changed the code that generates the light strength image so that it computes at each voxel the maximum light strength (over any view) that reaches the voxel, rather than the total light strength at that voxel.
A few experiments showed that this approach requires a very clean occupancy mask, so I ramped the blurred mask a bit more and used it instead of the unblurred mask in my usual occupancy-masked backprojection. Cells f1 through f4 show the resulting light strength images, cell f6 shows the max of these, cell g5 shows the reverse-video image, and the resulting geometry (with threshold 48) is shown to its right. For comparison, the occupancy mask is shown in cell d7. The geometry is not perfect, but it is very faithful to the occupancy mask. In other words, it is a near-optimal result for the given mask.
In retrospect, this pipeline isn't as strange as it might seem. I am merely computing the L-infinity norm at each voxel; if it was seen from any camera, then it is unoccupied. The only problem with this approach, or any approach that relies on maxes, is that it may not be robust. The slightest hole in the occupancy mask will result in a slash through the object's geometry.
The problem with this pipeline is that still involves a threshold. Contours generated by thresholding strong versus weak walls will be spatially shifted relative to their proper locations. A zero-crossing is still the better answer.
epi_masked_contour.tcl
epi_masked_contour.sht
I finally implemented myself (rather than trying to use vtk) masking of the zero-crossing by a thresholded occupancy mask. This works well! (The following description was worded for export to Bill Lorensen.)
Cell b1 shows the (blurred) flatland occupancy mask. Cell c1 shows the gradient of the occupancy mask along (blurred) light vector field lines. Mid-gray (128 on an 8-bit grayscale) represents zero, as usual. Cell a2 shows an Inventor-generated cross-section of the original polygonal scene, including color and texture. This is our gold standard.
Cell b2 shows (in bright yellow) the zero-crossing contours of cell c1, generated using vtk. Actually, a threshold of 122 (on an 8-bit grayscale, where 128 represents zero) was used to bias the generation of garbage contours toward the insides of the objects in the scene, rather than a mixture of insides, outsides, and random contours throughout empty space. A semi-transparent image of the Inventor-cross section is superimposed for comparison.
Cell c2 shows the result of clipping the contours in b2 by the occupancy mask in cell b1. This was performed (outside vtk) by deleting those polylines in cell b2 for which either endpoint, when mapped to the occupancy mask image (with bilinear interpolation of the latter), has an occupancy value of less than 32 (on an 8-bit grayscale). The SceneViewer in the upper-right corner shows the contours of cell c2 superimposed over a semi-transparent rendering of the original scene. The plane of the flatland light field is evident.
As expected, this clipping step almost completely eliminates the unwanted zero-crossings around the inside walls of the objects. Moreover, the remaining contours are nicely coincident with the polygonal edges of the gold standard cross section. This is by contrast to my previous experiments with thresholdings of shadow maps, etc. (see epi_contour*), in which the contours are slightly spatially displaced from their correct locations.
The main artifact of this pipeline appears to be a slight rounding of the corners of sharp objects, probably due to taking gradients along flow lines in a blurred vector field. This artifact will, however, scale down with pixel (voxel in 3D) resolution. The only other artifacts are those due to flaws in the given occupancy mask, such as weak walls in heavily occluded regions and lack of any walls on untextured objects. The former artifact should lessen in 3D. The latter won't.
Ideally, I would like to perform the polygonal clipping inside vtk so that I can conveniently follow it by largest-connected-component finding, Taubin smoothing, etc.
epi_back_tex.tcl
epi_back_tex.sht
Simple unfiltered backprojection (cell d5) and filtered backprojection (cell c5) of standard advanced scene without and with texture, mainly for reference. I don't seem to have these images recorded in any other convenient place.
epi_unfiltered_grad.tcl
epi_unfiltered_grad.sht
The usual gradient operator on the standard advanced scene with texture, using
filtered (left column of cells) and unfiltered (right column) backprojection.
No zero clamping was applied to the filtered backprojection. The two blowups
are, top to bottom, of cells g8 and n8 respectively. I seem to have never
tried this before. It confirms that the gradient operator performs better on
filtered backprojections despite their artifacts.
epi_silhouette.tcl
Addition of a shape-from-silhouette constraint. Cell c5 shows the filtered
backprojection of the standard advanced scene with texture, and cells g8 and e7
show the standard raw occupancy mask and ramped and blurred mask, respectively.
Cell e5 shows a shape-from-silhouette mask obtained by backprojecting a scene
consisting of unshaded white objects on a black background (visible in cell d1)
and taking the min color in a voxel instead of compositing. Note that the
front and back slabs are mirror images of one another, and their
backprojections are identical to one another, and similarly for the left and
right slabs. Cell d5 shows a blur of cell e5. Cell d7 shows the product of
the two blurred masks.
Combining this experiment with a recent attempt to confirm the results of
epi_reproject2, cell f5 shows an occupancy-masked
and visibility-masked backprojection using the new occupancy mask (cell d7),
and cell n8 and l7 show raw and blurred occupancy masks obtained by
running gradient operators on this new backprojection.
In general, applying a shape-from-silhouette constraint clears out all the
noise in some of the regions outside the objects - which regions depends on the
vageries of occlusions. As in previous experiments, computing an occupancy
mask from a masked backprojection clears out some of the noise inside the
objects, but also thins the walls. Thus, combining these two procedures
produces a mask that is thin, but clear of noise both inside and outside.
Note, however, the slot created above the central cube (cell d7) by the
shape-from-silhouette algorithm. This sort of discontinuity is a danger of any
nonlinear operator (like the min function used to compute the silhouette). The
corresponding part of Brian Curless's system is carving-from-backgrounds, and
it creates handles - the 3D version of a slot.
epi_silhouette
September 2, 1997
epi_silhouette.sht
Printed down to here for Shape From Light Fields notebook,
September 4, 1997
© 1997
Marc Levoy
Last update:
April 19, 2001 12:34:40 AM
levoy@cs.stanford.edu