Some years ago I saw the movie "A Scanner Darkly." There isn't much to talk about as much as the plot goes, but the visuals of the movie were unique. The movie was done with real actors but it the look of everything was as though it were drawn like a cartoon.
I thought about making an application that would allow some one to produce a similar effect in real time (or close to it) using a phone's camera. I thought I would be able to implement it with the K-means algorithm operating within color space (I will do another post on the details of this). Before diving into this task I needed to make sure that the phone was capable of doing this. I started by taking a look at Windows Phone and these were the main things that I needed to be able to satisfy:
- Is real time access to the camera available
- Can I render video frames to the screen at a rate
- Can the phone provide the computational capability to quickly do the image processing
One of the new capabilities that comes with the Mango update to Windows phone is access to the camera. In addition to getting information from the camera through tasks (which was available with the initial release of Windows Phone) Microsoft has granted developers the ability to paint a surface with a live feed from the camera, capture a video from the camera, capture a frame from the preview buffer, and take a photograph (without user interaction) from the camera. Let's examine how each one of those features does or does not contribute towards my goal and the program design.
Because of the the nature of my goal (to work with video) the Windows Phone Tasks (Camera Capture and Photo Chooser) won't work for my program. They both require user interaction for each frame captured. That's no way to work with video.
What about taking pictures automatically? This doesn't quire work either. Picture taking is slow. In general you'll find that the CCDs used in many digital devices are now able to capture and transmit the information from a full resolution photograph as quickly as they do when sending lower resolution video.
The ability to display the video buffer on screen looks promising. With it you can display what ever the camera sees. However this capability is only for displaying the camera's "vision" on the screen and rendering over it (such as in augmented reality).
This leaves two methods left: using the preview buffer and using the phone's video capturing abilities. Using the phone for video capture gives the highest framerate but it ceases to be real time. I'd be fine with that. That would just mean that some one would need to film a video and then it would play back with the video affect applied. But that would also require that I decode the resulting MP4 video myself (there's no video codec available to do this). So the preview buffer seemed like the best option. So I did a quick test to see how many frames I could capture per second (before performing any processing).
public MainPage()
{
InitializeComponent();
_camera = new PhotoCamera();
_camera.Initialized += new EventHandler<CameraOperationCompletedEventArgs>(_camera_Initialized);
videoBrush.SetSource(_camera);
}
void _camera_Initialized(object sender, CameraOperationCompletedEventArgs e)
{
var x = _camera.PreviewResolution;
int pixelCount = (int) (x.Width*x.Height);
buffer = new int[pixelCount];
_camera.PreviewResolution
Dispatcher.BeginInvoke(() => { });
Thread ts = new Thread(new ThreadStart(GrabFrames));
ts.Start();
}
void GrabFrames()
{
_camera.GetPreviewBufferArgb32(buffer);
var startDate = DateTime.Now;
for(int i=0;i<100;++i)
{
_camera.GetPreviewBufferArgb32(buffer);
}
var endTime = DateTime.Now;
var delta = endTime.Subtract(startDate);
}
The results I got back on a Mango Beta HD7 worked out to 10 frames per second. Not quite real time video. So it looks like my best option is to go with the MP4 video recorder. I'll have to figure out how to read frames from an MP4 file.
I'm glad I was able to figure that out before writing a substantial amount of code or doing a substantial amount of design.
Tags: Windows Phone 7, Development