Projecting the rendering of a computer model onto a statue

Marc Levoy
Stanford University
September 8, 1998

This web page documents some experiments I did recently using video projectors to illuminate statues with aligned computer renderings of the same statue. Why would you want to do this? Well, look at this example:

1	2

Figure 1 shows a photograph of the Stanford Happy Buddha statue illuminated with spotlights from the front and side. Figure 2 shows a photograph of the effect of projecting computer renderings onto the statue from the same directions. These renderings, made from a 3D model of the statue, consist primarily of an accessibility shading of the model geometry. They are similar to renderings made for the 3D fax project.

The visual effect of figure 2 to an observer standing in the room is of an antiqued statue, fully three-dimensional, and illuminated from the front and side. In fact, if the room is otherwise dark and if the projectors are well aligned, no amount of close examination will allow an observer to discover that the statue is not colored as it appears.

Some background on the idea

While traveling through France this summer ('98), I attended a sound and light show (Son et Lumiere) at the Blois castle in the Loire Valley. These nighttime spectacles, for which the French are famous, consist of telling the history of an architectural monument while manipulating the illumination of the monument. These shows are scripted in advance and controlled automatically.

The castle at Blois is well-known for its sound and light shows. To make this production, the artists took photographs of two walls of the interior courtyard of the castle, modified the photographs in some way, or turned them into line drawings, and then projected them back onto the walls. By placing the projector exactly at the camera's viewpoint, and by carefully aligning the image with the architectural features of the castle, its walls are made to appear differently colored, or differently decorated, or even as a cartoon of a castle. Since the walls are ornate and therefore naturally three-dimensional, they act like an 80-foot tall autostereoscope 3D display surface. The effect was startling.

Although this effect was probably accomplished without the aid of computers, it occurred to me that a similar effect could be achieved by scanning an object (such as a statue) to create a computer model, rendering the model with unusual surface treatment or illumination, then projecting the rendering back onto the object. In this case, the projector must be aligned with the projection matrix used when computing the rendering.

One advantage of using computer graphics for this task is that once the computer model has been created, any number of projectors may be employed, each displaying a rendering from its own viewpoint. It must have been extremely difficult for the artists at Blois to hand-modify multiple photographs of an object well enough that the modifications aligned among the views when projected onto the castle. Indeed, only certain types of modifications could be attempted by hand. Another advantage of using graphics is that animations can be computed and projected onto the object, for example in synchrony with a rotation of the object.

Note on previous work (added April 13, 2000): The use of projectors to change the appearance of an object has appeared in many guises beyond the Loire Valley. The 3D animated faces at Disney's Haunted House are one example. Henry Fuchs of the University of North Carolina has investigated some of these ideas in the context of his Office of the Future, in particular the notion of projecting images onto known irregular surfaces. His graduate student Ramesh Raskar, who worked on that project, has taken these ideas further. In work concurrent to or slightly later than my own, he experimented with projecting images onto a vase and a wooden model of the Taj Mahal, the 3D shapes of which he knew. His experiments produced some nice visual effects, which he describes in this technical report (subsequently published in Proc. Eurographics Rendering Workshop 2001).

How it is done

3	4

Figure 3 shows the rendering created for projection onto the side of the statue, and figure 4 shows the rendering created for projection onto the front of the statue. Due to the limited availability of projectors for this ad-hoc experiment, projectors from two different manufacturers were employed, a Sony for the side image and a Davis for the front image. The difference in minimum field of view between the two projectors accounts for the different sizes of the rendered statue in the two images.

5	6	7	8

Figure 5 shows a closeup of the statue as lit by uniform illumination coming from the front and side. This illumination was actually produced by the two video projectors both displaying flat gray fields. Figure 6 shows the statue as illuminated from the side by the Sony projector displaying figure 3, figure 7 shows the statue as illuminated from the front by the Davis projector displaying figure 4, and figure 8 shows the statue as illuminated by both projectors. These photographs were taken by an Olympus DL-600 digital camera with automatic exposure control. This accounts for the inconsistent brightness of the figures and partially accounts for their inconsistent color balance. However, the two projectors did have different color balances, an issue discussed in more detail below.

Some technical issues

1. The rendered images must be of high resolution, or they look poor when projected onto a real object. Video projectors were used in the present experiment, and their optics limited the resolution with which images could be projected onto such a small object as our buddha statue. Figures 3 and 4 show the resolution of our projected images. That this experiment succeeded at all is due to the fortuitous selection of a rendering style - accessibility shading - that does not depend on high-frequency detail.

2. The projectors must be well matched, calibrated and aligned. We used different projectors with strongly different color characteristics. Fortunately, the eye readily ascribes such differences to the illumination conditions and therefore does not find this error objectionable. However, failure to properly calibrate and align the projectors is fatal to the success of this illusion. Many hours were spent at this task, and the task was made more difficult by using a copy of the statue that differs from the version scanned. As an example of the importance of proper alignment, slight errors in the aiming of our Davis projector are visible along the the left edge of the amulet in figure 7.

3. Animation sequences must be smooth and well synchronized to the motion of the real object. The slightest missynchronization between object motion and image sequence playback causes the projected features to wander across the object. Moreover, the slightest hiccup in the playback rhythm, or the use of frame doubling to slow down the playback, causes beating between features of the renderings and the object.

4. The real object must be gray. If the object is too dark, it will be impossible to project an image onto it. Radiometric correction may be possible within some range of reflectivity, but this has not been tried. If the object is too bright, interreflections will pollute the image being projected. Since light cannot be subtracted, there is no way to correct for this problem.

5. The real object must have a matte finish. If the object is specular, as is the buddha, it will reflect the projector light in a view-dependent manner. Whether this reflection ruins the illusion or not depends on the illumination used when rendering the model and on the rendering style employed. In figures 3 and 4, the local key light in the rendering was placed slightly above the front projector, and a directional fill light of a different color was made roughly coincident with the direction to the side projector. Thus, the presence of accidental specular highlights was more or less comensurate with the apparent sources of illumination, and the illusion was preserved. See the next section for a further discussion of specular highlights.

6. When using multiple projectors, renderings must be correctly feathered where their projections overlap. Irradiance on a curved surface falls off with the cosine of the angle of incidence, for example falling to 0.707 for a surface point whose normal makes a 45 degree angle with a vector to the projector. If a second projector is placed at right angles to the first, the sum of their images at this point will be 2 x 0.707 = 1.414, which is too bright. Fortunately, angle-dependent feathering can be readily incorporated into most rendering systems - a RenderMan shader in our case.

7. The number of projectors needed to completely cover an object depends on its geometry. For a sphere, I believe that four projectors suffice. For non-convex objects, the number depends on the configuration of occlusions, which cause shadowing of the projected images. In theory, an infinite number of projectors may be required. I suspect that three will suffice for most frontally-viewed statues, including one positioned above the viewer, and five for statues viewed in the round.

Games with specular highlights

9	10	11	12

Although specular reflections from the surface of the real object are typically unwanted, and they may even destroy the visual illusion, as discussed above, they can also be used to advantage in certain circumstances. In figures 3 and 4, the projected images are neutrally colored, hence the specular highlight appears white, and the statue appears to be made of plastic. In figures 9 and 10, the statue base is given a purple cast. This causes the specular highlight to appear colored, producing a strong illusion that the base is made of metal.

A comparison of the statue base under neutral and colored illumination are shown in figures 11 and 12, respectively. Unfortunately, the Olympus camera operating under ambient illumination was not sufficiently sensitive to capture the illusion, which encompasses the entire base of the statue. However, these photographs at least serve to show the change effected in the color of the specular highlights. Note the difference between the colors of the two highlights in each photograph. This is caused by the differences in color temperature of the two projectors, but as previously noted, the eye ascribes this difference to illumination conditions and the illusion is not destroyed.

What's the killer app?

For statues, here are some random ideas:

• Rendering a Michelangleo statue in his highly sculptural painting style, as exemplified by the Sistine ceiling, then projecting this painting into the statue. Using data acquired during the Digital Michelangelo Project, we could turn the David into a 20-foot tall 3D painting!

• Projecting conjectured coloration onto Greek statues, which were typically originally painted. The projection could be turned on and off by the museum-goer at the press of a button, turning the viewing of sculpture into a hands-on experience.

• "Visual cancelation" of existing patina, dirt, or other coloration, for example to show how a statue will look after restoration.

• Virtual reconstruction of bas-reliefs (or maps of ancient Rome?), frescos (the ruined ceiling at Assisi?), or paintings (the Last Supper?).

• Paint-it-yourself! (virtually)

Returning to the architectural theme of the Son & Lumiere shows, and using our Cyra scanner to acquire the 3D model, one could imagine creating a sound and light spectacle for the Stanford quadrangle (or Memorial court), featuring Leland Stanford presiding over the driving of the Golden Spike, the collapse of the church in the earthquake of 1906, or perhaps Stanford as a picturesque ruin in 2000 years.