By Georg Petschnigg, June 2002

This is a detailed description of the mask creating and correction step. Go back here.

Steps	Detailed Description
1. Take the best non flash image given the current lighting conditions with less than 1/30sec exposure.	This image was taken using a Nikon CoolPix 995 camera with the following image settings: CAMERA : E995V1.6 METERING : MATRIX MODE : P SHUTTER : 1/7sec APERTURE : F2.6 EXP +/- : 0.0 FOCAL LENGTH : f8.2mm(X1.0) IMG ADJUST : STANDARD SENSITIVITY : ISO800 WHITEBAL : AUTO SHARPNESS : OFF DATE : 2002.06.02 22:12 QUALITY : FULL FINE SATURATION : 0 FOCUS AREA : CENTER Note the exposure time was more than 1/30sec, because the Nikon995 currently does not support an ISO1600 or above mode. The Nikon995 is the best one can currently do with a compact camera ISO wise. Still the image exhibits all the noise artifacts one would expect from a high ISO settings image.
2. Take the best flash image given the current lighting conditions. Steps 1 and 2 should not take longer than 1/30sec to avoid blurring and image registration.	The flash image shows the redeye problem I will attempt to correct. CAMERA : E995V1.6 METERING : MATRIX MODE : P SHUTTER : 1/60sec APERTURE : F2.6 EXP +/- : 0.0 FOCAL LENGTH : f8.2mm(X1.0) IMG ADJUST : AUTO SENSITIVITY : AUTO WHITEBAL : AUTO SHARPNESS : AUTO DATE : 2002.06.02 22:12 QUALITY : FULL FINE SATURATION : 0 FOCUS AREA : CENTER
3. Blur the non flash image using a 8X8 Gaussian.	Non flash image red channel before blurring.
	Blurred non flash image's red channel. This step is done to remove noise and to prevent errors if images are not aligned perfectly.
	As a reference, this is what the non flash image's red channel looks like.
4. Convert the blurred non flash image into  YCrCb color space. Before the conversion the RGB values are raised to the power 1/3 to accentuate the red eye. This step is due to [Patti98].	Non flash image's Cr component. noflash_ycbcr_image = rgb2ycbcr(img_noflash .^ 0.3);
5. Repeat step 4. for the flash image	Flash image's Cr component. flash_ycbcr_image = rgb2ycbcr(img_flash .^ 0.3);
6. Subtract the non flash Cr channel from the flash Cr channel and exit if no significant differences are found	This is the result of subtracting the non flash Cr image from the flash Cr image. The results of this operation lie in the range -1 to 1. This image only shows the positive parts scaled to the range 0 to 1. This is where the flash image was "more red" than the non flash image. The red eye lies at the maximum difference.  mask = flash_ycbcr_image(:,:,3) - noflash_ycbcr_image(:,:,3); At this point the algorithm could return if no "significant" Cr differences where detected, suggesting no red eye is present in the image. Currently the system does not detect the "absence" of redeye and expects at least one to occur. Microsoft Research in China developed a neural networked redeye detector that could be used to verify if redeyes are present.
7. Normalize the intensity  range to 0 to 1	This image shows the result of the subtraction normalized to the 0 to 1 range. The bright areas around the ear are due to image misalignment. This image shows JPEG artifact blocking which hamper the correction performance as well. mask = mask - min(mask(:)); mask = mask / max(max(mask));
	The difference image's histogram shows that the majority of the image is centered about the mean. The red eye is located at the extreme point on the right, where the Cr channel difference is  the largest.
7. Threshold the image against a lower bound and segment 8-connected regions into blobs	The lower threshold image estimates the largest redeye spread. The lower threshold is computed as follows: TLower = max([ 0.4 mean(mask(:)) + var(mask(:)) ^ 0.5]); The lower threshold is determined by adding the root of the variance to the mean difference. As a sanity check this number should be at least 0.4, a bit lower than the expected value of 0.5. Finally all 8-connected points are grouped into colored blobs, which is accomplished via the Matlab function bwlabel described here following Haralick, Robert M., and Linda G. Shapiro. Computer and Robot Vision, Volume I. Addison-Wesley, 1992. pp. 28-48. low = bwlabel(mask > TLower, 8);
8. Compute an upper threshold	The upper threshold is defined to be anything above six times the squared root of the variance plus the mean, This should assure that only statistically significant differences are included. At most thought the upper threshold should be 0.8, which was determined by looking at histogram distributions of the sample data set.    TUpper = min([0.8 mean(mask(:)) + 6 * var(mask(:)) ^ 0.5])
9. Keep only lower threshold blobs that contain pixels from the upper threshold image	This image shows the labeled blobs that contain larger Cr differences. Each blob is assigned a unique label, indicated in this image by color. Obviously there are some incorrectly included blobs as well, which will be removed in the geometry segmentation step. high = mask > TUpper; [r, c] = find(high); final_candidates = bwselect(low,c,r,8);low = bwlabel(low, 8);
10. Perform geometric selection using blob with largest Cr difference as seed	For each selected blob compute its Area, Minimum Bounding Rectangle and Eccentricity. Area is a good indication of size, the miminum bounding rectangle will be used to fit an ellipsoid, and the eccentricity gives a measure of how circle or line like a blob is. Eccentricity is described in detail here. The corresponding Matlab call is imfeature.  Finally I use the assumption that a redeye is located where the largest Cr difference occurs to find one seed eye. I use the seed eye's statistics to eliminate too large or non ellipsoid blobs. In this image for example the seed eye has the following statistics:  Area: 181 Centroid: [226.5193 169.6022] BoundingBox: [215.5000 160.5000 22 20] MajorAxisLength: 25.4041 MinorAxisLength: 10.3848 Eccentricity: 0.9126 Orientation: 36.4705 ConvexHull: [14x2 double] ConvexImage: [20x22 uint8] ConvexArea: 261 Image: [20x22 uint8] FilledImage: [20x22 uint8] FilledArea: 181 EulerNumber: 1 Extrema: [8x2 double] EquivDiameter: 15.1808 Solidity: 0.6935 Extent: 0.4114 PixelList: [181x2 double] Only blobs that are about within 70% of the seed eyes size and have an Eccentricity of less than 0.75 are kept. Those numbers were determined using the data set I took.
11. Fit ellipsoids into the bounding box of remaining blobs	Using the bounding box information of each redeye I fit an ellipsoid over the eye. The current implementation does not account for redeye partially occluded by eyelids or which no axis aligned images. This is a flaw.
12. Feather the generated redeye mask	The final redeye mask. Feathering is performed using a 3X3 Gaussian kernel.
After the redeye mask is complete color correction can be performed
	The segmented red eyes using the mask above
1. Predict red channel by averaging the red and green channel of the red eye 2. If needed adjust the average luminance to a mid gray tone	Result of predicting the red channel by averaging the blue and green channel. This image then is raised to the power of gamma which is computed below. The effect is to scale the redeye median to about 0.1 - which seems reasonably dark. gamma = log(.15) / log(median_eye_luminace);
	As a reminder the redeye image.
3. Composite the flash image and fixed redeye using the OVER operator and the mask as an alpha map	The final corrected image.
Go back to overview