September 8, 1998
This web page documents some experiments I did recently using video projectors to illuminate statues with aligned computer renderings of the same statue. Why would you want to do this? Well, look at this example:
Figure 1 shows a photograph of the Stanford Happy Buddha statue illuminated with spotlights from the front and side. Figure 2 shows a photograph of the effect of projecting computer renderings onto the statue from the same directions. These renderings, made from a 3D model of the statue, consist primarily of an accessibility shading of the model geometry. They are similar to renderings made for the 3D fax project.
The visual effect of figure 2 to an observer standing in the room is of an antiqued statue, fully three-dimensional, and illuminated from the front and side. In fact, if the room is otherwise dark and if the projectors are well aligned, no amount of close examination will allow an observer to discover that the statue is not colored as it appears.
While traveling through France this summer ('98), I attended a sound and light show (Son et Lumiere) at the Blois castle in the Loire Valley. These nighttime spectacles, for which the French are famous, consist of telling the history of an architectural monument while manipulating the illumination of the monument. These shows are scripted in advance and controlled automatically.
The castle at Blois is well-known for its sound and light shows. To make this production, the artists took photographs of two walls of the interior courtyard of the castle, modified the photographs in some way, or turned them into line drawings, and then projected them back onto the walls. By placing the projector exactly at the camera's viewpoint, and by carefully aligning the image with the architectural features of the castle, its walls are made to appear differently colored, or differently decorated, or even as a cartoon of a castle. Since the walls are ornate and therefore naturally three-dimensional, they act like an 80-foot tall autostereoscope 3D display surface. The effect was startling.
Although this effect was probably accomplished without the aid of computers, it occurred to me that a similar effect could be achieved by scanning an object (such as a statue) to create a computer model, rendering the model with unusual surface treatment or illumination, then projecting the rendering back onto the object. In this case, the projector must be aligned with the projection matrix used when computing the rendering.
One advantage of using computer graphics for this task is that once the computer model has been created, any number of projectors may be employed, each displaying a rendering from its own viewpoint. It must have been extremely difficult for the artists at Blois to hand-modify multiple photographs of an object well enough that the modifications aligned among the views when projected onto the castle. Indeed, only certain types of modifications could be attempted by hand. Another advantage of using graphics is that animations can be computed and projected onto the object, for example in synchrony with a rotation of the object.
Note on previous work (added April 13, 2000): The use of projectors to change the appearance of an object has appeared in many guises beyond the Loire Valley. The 3D animated faces at Disney's Haunted House are one example. Henry Fuchs of the University of North Carolina has investigated some of these ideas in the context of his Office of the Future, in particular the notion of projecting images onto known irregular surfaces. His graduate student Ramesh Raskar, who worked on that project, has taken these ideas further. In work concurrent to or slightly later than my own, he experimented with projecting images onto a vase and a wooden model of the Taj Mahal, the 3D shapes of which he knew. His experiments produced some nice visual effects, which he describes in this technical report (subsequently published in Proc. Eurographics Rendering Workshop 2001).
Figure 3 shows the rendering created for projection onto the side of the statue, and figure 4 shows the rendering created for projection onto the front of the statue. Due to the limited availability of projectors for this ad-hoc experiment, projectors from two different manufacturers were employed, a Sony for the side image and a Davis for the front image. The difference in minimum field of view between the two projectors accounts for the different sizes of the rendered statue in the two images.
Figure 5 shows a closeup of the statue as lit by uniform illumination coming from the front and side. This illumination was actually produced by the two video projectors both displaying flat gray fields. Figure 6 shows the statue as illuminated from the side by the Sony projector displaying figure 3, figure 7 shows the statue as illuminated from the front by the Davis projector displaying figure 4, and figure 8 shows the statue as illuminated by both projectors. These photographs were taken by an Olympus DL-600 digital camera with automatic exposure control. This accounts for the inconsistent brightness of the figures and partially accounts for their inconsistent color balance. However, the two projectors did have different color balances, an issue discussed in more detail below.
Although specular reflections from the surface of the real object are typically unwanted, and they may even destroy the visual illusion, as discussed above, they can also be used to advantage in certain circumstances. In figures 3 and 4, the projected images are neutrally colored, hence the specular highlight appears white, and the statue appears to be made of plastic. In figures 9 and 10, the statue base is given a purple cast. This causes the specular highlight to appear colored, producing a strong illusion that the base is made of metal.
A comparison of the statue base under neutral and colored illumination are shown in figures 11 and 12, respectively. Unfortunately, the Olympus camera operating under ambient illumination was not sufficiently sensitive to capture the illusion, which encompasses the entire base of the statue. However, these photographs at least serve to show the change effected in the color of the specular highlights. Note the difference between the colors of the two highlights in each photograph. This is caused by the differences in color temperature of the two projectors, but as previously noted, the eye ascribes this difference to illumination conditions and the illusion is not destroyed.
For statues, here are some random ideas:
Returning to the architectural theme of the Son & Lumiere shows, and using our Cyra scanner to acquire the 3D model, one could imagine creating a sound and light spectacle for the Stanford quadrangle (or Memorial court), featuring Leland Stanford presiding over the driving of the Golden Spike, the collapse of the church in the earthquake of 1906, or perhaps Stanford as a picturesque ruin in 2000 years.