CS 478 - Computational photography (Winter 2011)
"Hello Camera" Assignment
Due February 1, 2012 at 2:00 p.m.
Introduction: This assignment is an exercise intended to introduce you to the Frankencamera API, NVIDIA's Tegra platform, and the Android SDK, and should get your feet wet and your brain rolling for the term project. You will implement an autofocus algorithm on the Tegra tablet, and in doing so, you will gain understanding of the challenges involved in interfacing with a camera on a mobile platform.
Historical Note: A previous iteration of this assignment was offered in Winter 2009, using Nokia N900. Several students came up with autofocus algorithms that performed better than the stock Nokia implementation!
Steps:
- Getting Started: How to get set up.
- Overview: Quick guide to the code provided.
- Tasks: Things to do.
Deliverables:
- A zipped archive of your code. A tarball of fcam directory will suffice.
- An electronic copy of your write-up. Plaintext e-mail is OK.
- Sign up for a 10-minute grading session for you to demonstrate the functionality of the camera application with your autofocus implementation.
Send the above materials to cs478.win12.staff@gmail.com in an email with subject "[CS478] Assignment #1 - John Doe", where you replace John Doe with your full name.
Getting Started
You will be developing for the Tegra tablet, which runs the Android OS. Currently we have tested development on Windows 7 64-bit, Ubuntu 64-bit (Koala and Ocelot), and Mac OSX.
Managing your Tegra tablet [PLEASE READ]: The distribution of the Tegra tablets is made possible by the generosity of NVIDIA. Please take good care of these devices!
- Before you turn it on, make sure that it is charged.
- Avoid putting personal information and data on these devices, and if you do, be sure to delete them before you return the device at the end of the term.
- You may register a guest account at wirelessguest.stanford.edu to obtain temporary wireless access for the Tegra tablet, should you desire. They last up to two weeks, and can be renewed indefinitely.
Below is the list of steps for getting started with the code skeleton we provide. Don't be alarmed at the sheer number of the steps, as the procedure needs to be performed only once. [SHOW/HIDE]
- Download the Tegra Android Developer Pack 1.0r4 (NOT Tegra Android Toolkit) for your OS (200+ MB) and install it. From now on, the location at which TADP is installed will be called $(TADP).
- Download the assignment code skeleton. The archive also includes FCam and other necessary libraries. Unzip it where you would like--we shall call this location $(FCAM). It should create a subdirectory $(FCAM)/fcam.
- Launch Eclipse ($(TADP)/eclipse/eclipse) and select workspace $(TADP)/nvsample_workspace. You do not have to choose this workspace, but this workspace already contains some NVIDIA SDK examples.
- Let us start a new project that will contain our assignment.
- Click "File" -> "New" -> "New Android Project"
- Select "Create project from existing source".
- Enter the project name, e.g. "HelloCamera".
- Click "Browse" and select $(FCAM)/fcam/android/packages/fcamerapro directory, which should have been created when you unzipped the code skeleton.
- Click "Next", and select API Level 11 (Android 3.0).
- Click "Finish". This should create the project in your workspace.
- The code skeleton contains both JAVA and C++ source. As such, you need to convert the project into a mixed Java/C++ project in order for Eclipse to compile everything smoothly.
- We are ready to run the project. However, we should make sure that the Tegra tablet is recognized by the OS., so that we may try running the executable on the tablet.
- Mac OSX
- Connect the Tegra tablet to your machine using the USB cable. Check that the tablet is recognized by running adb devices. The binary adb is located in $(TADP)/android-sdk-macosx/platform-tools. Note that this folder may be automatically added to your PATH environment variable. If not, consider adding it.
student:~$ cd ~/tadp/android-sdk-macosx/platform_tools // the path may differ for your installation
student:~/tadp/android-sdk-macosx/platform_tools$ ./adb devices
List of devices attached
15c000000000000 device
If you get a serial number as above, then your machine is recognizing the tablet. You may get ??????????? instead of the serial number. If so, run:
student:~/tadp/android-sdk-mac_x86/platform_tools$ sudo ./adb kill-server
student:~/tadp/android-sdk-mac_x86/platform_tools$ sudo ./adb start-server
* daemon not running. startig it now on port XXXX *
* daemon started successfully *
Run adb devices again to confirm that the device is recognized. If you are still having trouble, make sure that the tablet is awake and turned on with USB debugging enabled.
- Ubuntu 64-bit
- Create the file /etc/udev/rules.d/51-android.rules.
- Enter the following text into the newly created file. Remember to replace the placeholder text with your userid.
SUBSYSTEM=="usb", ATTR{idVendor}=="0955", MODE="0666", GROUP="insert-your-userid-here"
- Change its permission to be readable by all, e.g.
student:~$ sudo chmod a+r /etc/udev/rules.d/51-android.rules
- Connect the Tegra tablet to your machine using the USB cable. Check that the tablet is recognized by running adb devices. The binary adb is located in $(TADP)/android-sdk-linux/platform-tools. Note that this folder may be automatically added to your PATH environment variable.
student:~$ cd ~/tadp/android-sdk-linux_x86/platform_tools // the path may differ for your installation
student:~/tadp/android-sdk-linux_x86/platform_tools$ ./adb devices
List of devices attached
15c000000000000 device
If you get a serial number as above, then your machine is recognizing the tablet. You may get ??????????? instead of the serial number. If so, run:
student:~/tadp/android-sdk-linux_x86/platform_tools$ sudo ./adb kill-server
student:~/tadp/android-sdk-linux_x86/platform_tools$ sudo ./adb start-server
* daemon not running. startig it now on port XXXX *
* daemon started successfully *
Run adb devices again to confirm that the device is recognized.
- Windows 7
- Try running the project. It should launch on the Tegra tablet.
Overview
There are several major components at work when the code skeleton compiles and runs. If you are not familiar with Android development or JNI, this explanation may help.
- The Frankencamera API (FCam) is a C++ interface for camera control developed here at Stanford. The Tegra tablet implements this API, meaning that one can invoke the functions in the API to make the Tegra behave accordingly. The relevant files are found in $(FCAM)/fcam/src, but you will not have to modify these at all. (As such, they may not be visible in the Eclipse workspace.)
- The programming language of choice for developing on Android is JAVA (with Eclipse as the development environment.) src/com/nvidia/fcamerapro/FCameraPROActivity.java (in Eclipse) represents a main application (or "activity", in Android-speak.) You will find upon inspection that it contains some initialization code (onCreate(...)), but really no code whatsoever for camera control.
- The JAVA side includes some machinery for representing and dealing with images on filesystem (classes Image, ImageStack, ImageStackManager.) The JAVA side also includes UI components, like CameraFragment and CameraView.
- The JAVA class FCamInterface is a utility class that handles interaction with the C++ FCam API. Upon inspection, you will see that methods in this class invoke a corresponding native (C++) method, which are defined in jni/FCamInterface.cpp, either to retrieve parameters or to send parameters. If you look at the native methods being invoked, they simply create a message (of type ParamSetRequest) and append it to a work queue. The message is consumed in the work thread.
- The work thread in jni/FCamInterface.cpp basically runs an infinite loop. In each iteration of the loop, it consumes any messages in the work queue, appropriately sets camera settings, and asks the camera to apply them, and deals with images returned by the sensor.
- There is a file called jni/MyAutoFocus.h, which will include your autofocus implementation.
Check out the lecture slides on the codebase for a more visual overview of the codebase structure.
If you have never tried Android programming, feel free to try out the Hello World example from Android Developers site. (Scroll down directly to "Create a New Android Project".)
You should have successfully executed the skeleton code by now. It is a functional camera application, equipped with a viewfinder, user interface for changing settings, and it will save pictures into /data/media/DCIM/fcam in the tablet. (Use adb pull to upload the images onto your machine. See ADB documentation for usage.) Play around with the settings to observe what they do. Note that in order for you to appreciate the setting changes, it is best to set everything to manual first, as changing one setting might cause another setting to compensate for it, in case the latter is in automatic mode.
In theory, it is possible to complete the assignment by changng the following files only (though you may freely edit the others or add new files as well. Your task is described in detail in the next section.) These files have sections marked "TODO"---it is hard to miss them.
- jni/MyAutoFocus.h: You will write the "interesting" bit of the assignment here. The rest is for connecting the bits and pieces together.
- jni/FCamInterface.cpp: You will need to create a mechanism with which the JAVA-side can tell the C++-side to engage in autofocus. Much of it is in place already.
- jni/ParamSetRequest.h: Not absolutely necessary. However, you may want to define more message types here, instead of hard-coding them in class FCamInterface.
- CameraFragment.java: This is a class that represents the viewfinder. Because you will activate autofocus by tapping the viewfinder, you will need to include some instrumentation to react properly to those user-generated events.
- res/values/strings.xml: Adding new menu items can be done here, though we leave it to you to figure out how they are hooked into the application.
Tasks
- !!!! Read the section on tips and bugs before you start coding. !!!!
The provided code skeleton is a functional camera application, save the lack of an autofocus algorithm. There is already instrumentation, albeit incomplete, to connect the user event of tapping the viewfinder to the invocation of the autofocus routine that you will write. Follow the control flow, and complete the instrumentation as necessary. Most importantly, write the autofocus algorithm.
- TASK A (Autofocus): Populate jni/MyAutoFocus.h, and instrument the rest of the codebase to activate it appropriately.
- Hint: This is an exercise in designing a state machine.
- Hint: For this purpose, you most likely will have to write code that analyzes an incoming frame for its "focused"-ness. Examine the FCam API to learn how to access the raw pixel values from an incoming frame. See the tips below.
Once your autofocus is working reasonably well, we can implement touch-to-focus. Tapping the screen should not only activate autofocus, but also try to focus on the object the user has tapped.
- TASK B (Touch-to-focus): Alter your code appropriately so that tapping the screen will achieve touch-to-focus. That is, you should focus when the viewfinder image at or around the region touched is sharp. Note that this should not replace the default behavior, but should be added as an option in the UI under "Touch Action".
Many consumer cameras sport a number of scene modes (portrait, macro, sports, night, landscape, for instance) and specialize their internal algorithms to the user's choice of modes. Or, perhaps there are cases in which the stock autofocus will fail. Design a special autofocus algorithm for one of the following scenarios:
- Low-light: if the scene is really dark, and requires boosting the gain (and therefore noise), it may affect sharpness computation. Should the autofocus routine take this into account?
- Action: sports photography is difficult because actions happen in a split second. A robust autofocus algorithm that takes its time may cause the photographer to miss the moment. Is it possible to make the process lightning quick (at the cost of being less robust)?
- Color: This mode should always try to maximize focus on an object of particular color. How should the user specify the color in this case?
- TASK C (Custom mode): Implement one of the modes above as an extension, and add appropriate user interface for triggering it.
Write-up:
- Q1. Succinctly describe the autofocus algorithm you implement. A small paragraph or two should suffice.
- Q2. What is the failure mode of your algorithm, if any?
- Q3. Most likely, your algorithm is probably parametric (e.g. depending on some specific thresholds, termination criteria, range of lens motion, speed of lens sweep, et cetera.) How did you arrive at your current parameters?
- Q4. Succinctly describe what scene mode you implemented and how.
- Attachment: Submit two photos of the same scene (or two very similar scenes), one using your initial autofocus and the other using your scene mode. These should ideally demonstrate that your scene mode works as intended. You may submit two actual photographs, or two screenshots.
- Q5. How did you like working with the Frankencamera API? Were they intuitive? Sufficiently expressive?
- Q6. How did you like working with the Tegra development environment? (Tegra/Eclipse/ADB)
Tips & Bugs:
Use the logging utility to your benefit. Both JAVA and C++ sides can write to the log easily. See jni/Common.h for the macros on C++ side; use Log class on JAVA side. The log should show real-time on "LogCat" tab on Eclipse. If not, run adb logcat on console. This will save you lot of debugging time.
The FCam implementation on Tegra currently does NOT seem to support setting lens speed when moving it. It currently defaults to the maximum speed. If you call FCam::Lens::setFocus(...) (which you absolutely will), be wary of this fact, as the lens moves amazingly fast.
Devices that are attached to the sensor will automatically tag every frame with metadata. This tells you what settings were used to take a particular frame. For instance,
FCam::Frame f = sensor.getFrame();
fprintf(stdout, "The average focus setting during the frame: %f\n", f["lens.focus"]);
See here for the list of tags added by the lens. Other attached devices, if any, will add tags as well.
According to FCam specification, a shot request can ask for FCam::SharpnessMap of a given size. However, the specification does not guarantee that the request for a particular size will be honored. In fact, the current implementation on Tegra does NOT. Hence, you will have to write your own routine for evaluating sharpness. You can access the raw data of a frame by calling FCam::Frame::image(). The type of the image returned is YUV420p (as explicitly requested by FCamInterface.cpp---look in the code), which is defined in include/FCam/Base.h.
You can take a screenshot by going to Window -> Open Perspective -> DDMS in Eclipse. This will open up a "Devices" window, which has a button for capturing screenshot. Note: If the screenshot comes out blank, try disabling and re-enabling USB debugging on your tablet. This can be done by going to Settings -> Applications -> Development on your tablet.
The tarball of the code skeleton also contains a number of micro-examples that uses FCam, under directory examples. If you have trouble following the use of FCam, look at these first.
© 2011-2012 Jongmin Baek