-------------------------------------------------------------------------------

	  CS 448 - High-X imaging I: high resolution, April 18, 2002

-------------------------------------------------------------------------------

		    *** Introduction to high-X imaging ***

Enabling technology I: high sampling rate:
	o digital camera sensors will soon have more pixels than they need
		- Canon EOS D60 has 6.52 megapixels (3152 x 2068), = 35mm film
		- noise rises with resolution, since # photons stays constant
	o or one sensor with a high frame rate
		- El Gamal's programmable digital sensor
			-> photos of fan blade, Kleinfelder et al., ISSCC '01
			-> statistics about chip, ibid, p. 7
		- noise rises with frame rate, for same reason
	o or multiple sensors per camera
		- example is 3-chip color camera
		- hard to engineer, thus expensive
	o or sensing during aiming and focusing
		- requires image stabilization



Enabling technology II: image stabilization:

	o basic operation is finding corresponding features in adjacent images
		- find offset dx,dy such that norm( f1(x,y) - f2(x+dx,y+dy) )
		  over a window of x,y pixels in images f1 and f2 is minimized
		- same as finding the cross-correlation between two images
		- many variants, e.g. for treating multiple images at once
		- output is called the optical flow field

	o computational expense
		- larger window is more robust, but more expensive
		- can be computed per-block
			-- 16 x 16 pixels in MPEG motion estimation
		  or per-block, then interpolated to pixels, or
		  or per-pixel

		- robustness increases with increased frame rate
		  because objects move a shorter distance, and
		  intensities change less per frame

		- computational expense is independent of frame rate
		  because smaller search windows suffice to find objects

-------------------------------------------------------------------------------

			    *** High resolution ***

Film formats:
	o 35mm still photography
		o horizontal format on 35mm film stock
		o 36mm (wide) x 24mm (high) image
	o 35mm cinematography
		o vertical format on 35mm film stock
		o 22mm x 16mm image
	o Vistavision
		o horizontal format on 35mm film stock (like still photography)
		o 38mm x 25mm image
		o used for special effects
	o 70mm cinematography
		o vertical format on 65mm film stock
		o 48mm x 22mm image
		o limited use nowadays
	o IMAX and OMNIMAX
		o horizontal format on 65mm film stock (70mm print stock?)
		o 70mm x 49mm (IMAX) or 70mm x 50mm (OMNIMAX)
	o medium format still photography
		o 2 1/4 x 2 1/4 inch image
	o large format still photography
		o 4 x 5 inch image, or 5 x 7, 8 x 10, or 11 x 14

Film / computer graphics hybrids:
	o high-end film scanner
			-- Kodak RFS 3600
		- assumes 65 line pairs per mm (lpmm) for film (optimistic)
		  x 2 for Nyquist rate = 130 dots per mm (3300 dpi)
		  x 22mm x 16mm (35mm movie) = 2860 pixels x 2080 pixels
			-> Kodak photomicrographs of film grain
		- scanning of 4 x 5 film frame would be 13,208 x 16,510 pixels
	o CG animation for distribution on film
		- 2048 x 862 (Pixar's "A Bugs Life")
		- extrapolated to IMAX film format = 4000 x 1750 pixels

Digital still cameras:
	o high-end consumer
			-- Nikon Coolpix 995, about $900
		- 2048 x 1536 pixels (3.3 megapixels), color mosaic
		- about 6mm x 6mm (small!)
		- custom lenses
	o prosumer
			-- Canon EOS D60, about $3000
		- 3152 x 2068 pixels (6.5 megapixels), color mosaic
		- 23mm x 15mm CMOS sensor (larger wells, so lower noise)
		- compatible with standard 35mm SLR lenses

	o professional
			-- Sinarback 44 HR, $30,000
		- digital back for Sinar 4 x 5 inch view camera
		- 4080 x 4080 pixels (16 megapixels), color mosaic
		- 37mm x 37mm CCD sensor (roughly same size pixels as Canon)
		- thermoelectrically cooled for lower noise (14 bit pixels)
		- "Microscanning" moves sensor in 1/2 pixel increments,
		  producing 75 megapixel (?!) non-interpolated color,
		  for still scenes only
		- "Macroscanning" moves sensor through a 4 x 4 grid of tiles,
		  producing 80 megapixel (?!) images (450MB file!),
		  for still scenes only, requires stitching
			-> images from Sinar calendar, resolutions unknown

Video cameras:
	o National Television Standards Committee (NTSC), digital encodings
		- 640 x 480 pixels (square pixels)
		- 720 x 480 pixels (D-1 format, rectangular pixels)
	o high-definition television (HDTV)
		- 1920 x 1080 x 30fps interlaced
	o computer displays
		- VGA: 640 x 480 pixels, various refresh rates
		- SVGA: 800 x 600
		- XGA: 1024 x 768
		- SXGA: 1280 x 1024
		- UXGA: 1600 x 1200

	o the Stanford Multi-Camera Array
		- 16 x 8 array of VGA cameras arranged with abutting views
		  = 10,240 x 3,830 pixels
		  or 7,680 x 5,120 pixels if the cameras are turned sideways

		- 10,240 pixels / 360 degree panorama = 28 pixels per degree =
		  1,400 pixels for 50 degree FOV (almost matches Nikon Coolpix)

Texture synthesis techniques:
	o statistical techniques
			-> Heeger, Pyramid-based..., Sig95
		- a good representative of statistical synthesis techniques
		- analyze sample and noise using steerable filter banks, then
		  alternate matching histograms of:
			-- filter coefficients of noise to coefs of sample
			-- synthesized texture to original sample
		- works on stochastic textures, but poorly on structured ones

	o search techniques
			-> Efros and Leung, ...non-parametric sampling, ICCV99
		- for each pixel, search for similar neighborhoods in sample
		- very slow, but the basis for Wei and Levoy and others

			-> Wei and Levoy, Fast texture synthesis..., Sig00
		- based on Efros and Leung, two orders of magnitude faster

	o patch techniques
			-> Praun et al., Lapped textures, Sig00 (& cover)
		- manually specified patches laid down randomly on 2D manifold
		- works surprisingly well for some textures, extremely fast

			-> Efros and Freeman, Image quilting..., Sig01
		- Efros-Leung search for block + minimum-error cut betw. blocks
		- roughly as fast as Wei and Levoy and usually more effective

	o application to super-resolution
			-> Hertzmann, Image analogies, Sig01
		- combine the composition of one image with the texture
		  of another; latter might be of higher resolution

			-> Sawhney, Hybrid Stereo..., Sig01
		- combine high-resolution image with a low-resolution image
		  taken from a slightly different viewpoint to yield a
		  high-resolution image pair



-------------------------------------------------------------------------------

			     *** Sources cited ***

High sampling rate sensors:

	Kleinfelder, S., Lim, S., Liu, X., Gamal, A.,
	A 10,000 Frames/s 0.18 µm CMOS Digital Pixel Sensor
	with Pixel-Level Memory,
	Proc. 2001 International Solid State Circuits Conference.

Super-resolution:

	* Zomet, A., Peleg, S.,
	Super-Resolution from Multiple Images having Arbitrary Mutual Motion,
	in S. Chaudhuri (ed.), Super-Resolution Imaging,
	Kluwer Academic, September 2001.

Texture synthesis algorithms:

	Heeger, D.J., Bergen, J.R.,
	Pyramid-Based Texture Analysis/Synthesis,
	Proc. Siggraph '95.

	Efros, A., Leung, T.,
	Texture synthesis by non-parametric sampling,
	Proc. ICCV '99.

	Wei, L.-Y., Levoy, M.,
	Fast texture synthesis using tree-structured vector quantization,
	Proc. Siggraph 2000.

	Praun, E., Finkelstein, A., Hoppe, H.,
	Lapped textures,
	Proc. Siggraph 2000.

	Efros, A.A., Freeman, W.T.,
	Image quilting for texture synthesis and transfer,
	Proc. Siggraph 2001.

Super-resolution using texture synthesis:

	* Hertzmann, A., Jacobs, C.E., Oliver, N., Curless, B., Salesin, D.H.,
	Image Analogies,
	Proc. Siggraph '01.

	Sawhney, H.S., Guo, Y., Hanna, K., Kumar, R., Adkins, S., Zhou, S.,
	Hybrid Stereo Camera:
	An IBR Approach for Synthesis of Very High Resolution
	Stereoscopic Image Sequences,
	Proc. Siggraph '01.

-------------------------------------------------------------------------------