In fact, if you can come up with an excellent algorithm, it may just become the default implementation provided in the Frankencamera API, or in Nokia N900 in the future!
Getting Started
Developing for N900 will require a cross-compilation environment, such as Scratchbox. We have created a vanilla virtual machine (loaded using VMWare Player or Fusion) that has everything installed. You can obtain the VM by the following means:
- Downloading (Warning: 2.4 GB zipped / 6.1 GB unzipped)
- Visiting Rm. 360 for a direct transfer.
- Visiting Rm. 360 to check out an installation disk (a DVD containing a tarball.)
[Not recommended] If you prefer to create your own VM, or install directly on a physical machine, follow these instructions.
If you are enrolled in the course, you should have received an N900 which has already been
configured with the necessary modules and updates. You can gain root privileges by entering root (password: root) in a terminal.
Managing your N900 [PLEASE READ]: The distribution of the N900's is made possible by the generosity of Nokia. Please take good care of these devices!
- Before you turn it on, make sure that it is charged.
- You may wish to confirm that the time/date setting is correct. (The time zone is most likely incorrect.)
- Your N900 will occasionally prompt you to update Maemo 5. In general, avoid system software update (individual packages are OK) as it may interfere with the kernel modules we have installed.
- Avoid putting personal information and data on these devices, and if you do, be sure to delete them before you return the device.
Tips for using your N900:
- Jongmin is hosting a guest account on the Stanford network (username cs448a-n900 and password password) for the duration of the assignment, for your convenience.
- ssh is your friend: N900 has the best-of-class screen size and resolution, and a physical keyboard, but it probably does not come close to a full 100+-key keyboard and a huge monitor. Also, ssh allows you to read debug outputs while you are running applications.
- You will most likely need to have your N900 connected to the internet via wi-fi. If you do set up connections, turn off power saving via Settings > Internet Connections > Connections > Edit > Advanced. This prevents ssh connections from being dropped. Unfortunately, this will drain the battery faster, so remember to keep it charged.
- It's probably useful to add shortcuts to your desktop (for terminal, settings, et cetera.)
Your N900 should already have ssh installed. To find out the IP address of your N900, you may either visit sites that tell your IP, such as this, or open an X-terminal and do the following:
~ $ root
Nokia-N900-XX-XX:~# ifconfig
Now scroll up and find the field "inet addr". Once you know your IP address, you can log onto your N900 remotely from your own machine.
cs448a@ubuntu:~$ ssh root@XXX.XXX.XXX.XXX
Nokia-N900-XX-XX:~#
You may now login to Scratchbox as follows:
cs448a@ubuntu:~$ scratchbox
[sbox-FREMANTLE_ARMEL: ~] >
Scratchbox is a chroot environment, meaning that the root directory has been mounted from the filesystem. The path to this directory in the VM itself is given by the symbolic link called scratchbox-root in your home directory of the VM. Likewise, the home directory in Scratchbox is pointed to by a symbolic link called scratchbox-home. Therefore,
[sbox-FREMANTLE_ARMEL: ~] ls
and
cs448a@ubuntu:~$ cd scratchbox-home
cs448a@ubuntu:~/scratchbox-home$ ls
will return the same results. This is useful because Scratchbox does not come with a good text editor. So you will most likely be editing file on cs448a@ubuntu:~/scratchbox-home in your VM.
Running binaries built in Scratchbox is now easy. scp the file onto your N900 from Scratchbox, and run the binary from an ssh terminal. For instance (hypothetically), in Scratchbox,
[sbox-FREMANTLE_ARMEL: ~] > g++ myapp.cpp -o myapp
[sbox-FREMANTLE_ARMEL: ~] > scp myapp root@XXX.XXX.XXX.XXX:/home
[Note] Try to put your files in /home, as there is only limited space in the root file system. You may now ssh into your N900, change directory to /home and run your binary:
cs448a@ubuntu:~$ ssh root@XXX.XXX.XXX.XXX
Nokia-N900-XX-XX:~# cd /home
Nokia-N900-XX-XX:/home# ./myapp
You could, of course, open an X terminal on your N900 directly, but you might prefer typing on a full keyboard and reading the console output on a full monitor.
Hello World!
If you have completed setting up your N900, this step is relatively easy. Write a very simple command-line program, like Hello World. Compile your code in Scratchbox, transfer the binary onto your N900, and confirm that it runs.
There are a couple of Qt examples in sample directory of Scratchbox. If you built your own Scratchbox, you can find these files here. (They are from Nokia's official Qt examples.) To build these, run qmake -project in the appropriate folder, followed by qmake XXXXXX.pro and make, where XXXXX is the name of the .pro file that resides in the respective folder. This will generate an executable of the same name. For instance, the following would work:
cs448a@ubuntu:~$ scratchbox
[sbox-FREMANTLE_ARMEL: ~] > cd sample/analogclock
[sbox-FREMANTLE_ARMEL: ~/sample/analogclock] > qmake -project
[sbox-FREMANTLE_ARMEL: ~/sample/analogclock] > qmake -unix analogclock.pro
[sbox-FREMANTLE_ARMEL: ~/sample/analogclock] > make
[sbox-FREMANTLE_ARMEL: ~/sample/analogclock] > scp analogclock root@XXX.XXX.XXX.XXX:/home
Take a minute to familiarize yourself with basic Qt. You will presumably need some minimal experience with Qt to build a reasonable interface to whatever application you make, but no more than what is involved in the examples, unless you intend to use other features, such as multi-threading, event handling via signals and slots, networking, et cetera. A set of examples and their source code can be found here. There is a more serious tutorial from Nokia, released two weeks ago. They would appreciate any feedback, in fact!
"Two-Focus"
TIP for TASK (I): Given two taps by the user, finding d_1 and d_2 is not an easy problem. It is called in literature "depth from focus" or "depth from defocus." An alternative approach is to choose a focus setting that makes the two objects equally sharp. This would fail if the two objects have different amount of texture, however, so you may want to focus on them individually to determine how sharp each of them is when in focus, and use this information as a normalizer. If you have other ideas, feel free to discuss them with the staff and/or implement them. We honestly do not know how well it will work on the N900, so it is not imperative that it works well, but be sure to approach the problem with rigor.
NOTE: Task (I) and Q7 are optional for singleton teams.
Write-up:
- Q6. Show the derivations in the first task.
- Q7. If you do task (I), describe your algorithm. If it succeeds, argue why it does. If it does not, explain why it does not.
Hints:
You may implement the features in separate applications, if you find it more convenient.
You may consider creating a MyAutoFocus class which supports function calls like initialize(), terminate(), getBestFocus(), processFrame(FCam::Frame::Ptr f), et cetera, so that you may aggregate all your edits into your own file(s).
As you can figure out from the two button widgets, you can set the focus of the lens by calling FCam::Lens::setFocus(...). For instance,
m_lens.setFocus(m_lens.farFocus(), -1); // focus as far as possible
m_lens.setFocus(m_lens.nearFocus(), -1); // focus as close as possible
See FCam/Lens.h to learn what the two arguments do. This header also contains other useful function calls to the lens object.
Remember that moving the lens is not instantaneous. (In fact, you might even want the lens to slow down and not move at its maximal speed.) If you call FCam::Lens::setFocus(...), it might be a while until you get back a frame with the desired setting. The sensor will continue streaming while you are moving the lens!
Devices that are attached to the sensor will automatically tag every frame with metadata. This tells you what settings were used to take a particular frame. For instance,
FCam::Frame::Ptr f;
f = sensor.getFrame();
Lens::Tags * t1 = f->tags(&m_lens);
fprintf(stdout, "The average focus setting during the frame: %f\n", t1->focus);
(See the documentation for a list of available parameters.) If you find that the frame you have just read turns out to be sharp, you might want to set the focus setting of your snapshot to be the same as this one.
N900 has hardware support for measuring sharpness in parts of the image. This is already activated in the code as follows:
m_shot.sharpness.enabled = true;
m_shot.sharpness.size = FCam::Size(16, 12);
This effectively creates a 16x12 grid of autofocus points. You can query the sharpness "score" of each region by doing the following:
FCam::Frame::Ptr f;
f = sensor.getFrame();
fprintf(stdout, "%d x %d\n", f->sharpness.size.width, f->sharpness.size.height); // should print out 16 x 12
fprintf(stdout, "Sharpness score at the top-left corner: ", f->sharpness(0,0) >> 10);
This is the recommended method of measuring sharpness, as dealing with raw pixels in software is not as fast.
A naive algorithm might go from the nearest focus to the farthest focus in some number of steps, asking the sensor to continuously stream, and calculate some score on each frame that comes back, remembering the focus setting of the best frame. Of course, this is not particularly robust...
You can access the raw data of a frame by calling f->image.data. It returns, by default, an image in 8-bit UYVY format; in each row, every odd-numbered pixel will have U and Y channel available, and every even-numbered pixel will have V and Y channel available. Finally, the raw data is in a row-major form (a 1-dimension representation in which every row has been concatenated together. Below is a code snippet for a simple grayscale conversion:
unsigned char * myimage = new unsigned char[width*height];
for (i=0;i<height;i++) {
for (j=0;<width;j++) {
myimage[i*width+j] = f->image.data[(i*width+j)*2+1];
}
}
Currently pressing the shutter takes a 5MP shot and saves it as a JPEG file in /home/user/MyDoc by calling FCam::AsyncFileWriter::saveJPEG(...). However, the color balance is not very good currently. If you want, you can replace this with FCam::AsyncFileWriter::saveDNG(...) to save the RAW data instead as a DNG file, and convert them using dcraw or Adobe Camera Raw into 8-bit formats.
You can grab the screen by pressing Ctrl-Shift-P on your N900. Unfortunately, as MyViewfinderX draws directly on screen, its content will not be captured. The screenshots are saved in /home/users/MyDoc/.images/Screenshot.
[Advanced] N900 features an ARM Cortex-A8 processor, which has a NEON SIMD unit. If you are doing serious pixel math (like convolution), then consider using NEON intrinsics for speedup. x4 improvement (over CPU) is typical, as long as you are not overloading the unit. This advice applies to the term project especially.
