Mixed Scale Motion Recovery

james davis

Abstract

Motion frequently occurs at mixed scales. Subtle localized motion often takes place together with long range movements. For instance, the pose of a soccer player is contained in a much smaller volume than the entire playing area, and the expression on an actor’s face is at a much smaller scale than the room in which the actor moves. Recovering these motions requires high resolution imagery suitable for resolving small details. It simultaneously requires increasing the working volume in which recovery is possible. Traditional motion capture has concentrated on single scale domains. This dissertation presents a framework for discussing mixed-scale motion recovery systems, and proposes a specific system design.

Mixed scale motion can be recovered with a foveated tracking system. Such a system has multiple stages. A wide area tracking subsystem makes use of many cameras to robustly localize motion. A small scale tracking subsystem makes use of narrowly focused cameras to recover motion with high accuracy. These latter cameras are mounted on pan‑tilt heads and can be dynamically oriented in the correct direction to provide high resolution coverage of the active region. This system is demonstrated on a specific application of practical interest to the entertainment industry, capturing simultaneous body and facial motion.

Documents

PhD Dissertation	[PDF 15MB] [PS 68MB]
Slides from oral defense	[PDF 3.2MB] [PowerPoint 4.1MB] [HTML]

Video Figures

	Motivational example from system Dissertation Figure 5 - Oral Defense Slide 9 [RealVideo 1MB] [Cinepak AVI 9MB] As a person moves through a room sized environment, body and facial motion are simultaneously recovered. (Bottom left) The recovered large scale body motion. (Bottom right) The recovered facial motion.
	Large scale feature integration Dissertation Figure 33 [RealVideo 2MB] [Cinepak AVI 16MB] [QuickTime14MB] Intersecting rays can be used to determine the three dimensional location of targets. (a-left) A 3D view of a set of camera observation rays for just one target. Colored dots are cameras mounted on the ceiling, and each line represents a camera observation. (b-middle) The set of observing rays for multiple targets. (c-right) The target locations extracted by intersecting these rays.
	Latency is corrected using prediction Dissertation Figure 50 - Oral Defense Slide 37 [RealVideo 1MB] [Cinepak AVI 31MB] This sequence shows the video stream from a pan-tilt camera as it is directed to follow a moving target. In the first row, the camera is directed without prediction. Note that the target often falls out of the video frame. In the second row prediction is used. In this case the target stays approximately centered in the video frame at all times.
	Tracking small scale features Dissertation Figure 55 [RealVideo 1MB] [Cinepak AVI 6MB] [QuickTime10MB] The small scale recovery system uses painted marks as features for motion recovery. A close-up of features on the face being tracked with a Lucas-Kanade style algorithm is shown here.
	Recovered small scale geometry Dissertation Figure 58 [RealVideo 1MB] [Cinepak AVI 1MB] [QuickTime1MB] Recovered 3D face geometry shown from several viewpoints.
	Comparing small scale motion to video Dissertation Figure 62 - Oral Defense Slide 41 [RealVideo 1MB] [Cinepak AVI ??MB] [QuickTime15MB] Still frames from a sequence of recovered facial geometry.
	Reconstruction of missing markers Dissertation Figure 67 - Oral Defense Slide 46 [RealVideo 1MB] [Cinepak AVI 17MB] A sequence of frames showing the quality of reconstruction as successively more feature points are occluded. Even with many occluded features, the reconstructed face remains believable.
	Video of simultaneous body and face motion Dissertation Figure 68 [RealVideo 1MB] [Cinepak AVI 21MB] [QuickTime20MB] Live action of simultaneous body and facial motion. The subjects large scale motion causes the sub-volume within which small scale facial motion occurs to traverse most of the global working volume.
	Recovered large scale motion Dissertation Figure 69 - Oral Defense Slide 30 [RealVideo 1MB] [Cinepak AVI 7MB] Recovered large scale body motion. The three dimensional motion of each target has been recovered, and is used here to visualize the overall motion of the subject.
	Recovered small scale motion Dissertation Figure 70 [RealVideo 1MB] [Cinepak AVI 7MB] Recovered small scale body motion. The motion of each facial feature was recovered, and the entire data set fit to a facial model. This model was used to reconstruct the motion of any occluded features.
	End - to - end review Oral Defense Slide 47 [RealVideo 2MB] [Cinepak AVI 33MB] Reviews the major components in the system.