Depth cameras go by many names: ranging camera, flash lidar, time-of-flight (ToF) camera, and RGB-D camera.  The underlying sensing mechanisms are equally varied: range-gated ToF, RF-modulated ToF, pulsed-light ToF, and projected-light stereo.  The commonality is that all provide traditional (sometimes color) images and depth information for each pixel (depth images) at framerate.  Existing commercial offerings, such as the Swiss Ranger SR4000 and PMD Tech products, currently cost ≈$10,000.   Thus, I'm extremely excited by Dieter Fox's recent statement about a sub-$100 depth camera that could hit stores later this year!  Dr. Fox has already leveraged a similar (this?) sensor to build cool 3D SLAM maps akin to Google Street View indoors -- see videos below.  Is Dr. Fox's employer (Intel) building depth cameras?  Is this a new PrimeSense offering?  Or could it hail from fellow Seattle powerhouse, Microsoft, who not long ago purchased 3DV Systems (purveyor of ToF cameras) and who plans to release Project Natal (rumored to be projected stereo) later this year for the XBox 360?  I'd love details, but am intrigued regardless!  Updated March 31st 2010:  Big news -- PrimeSense is supplying the 3D sensing technology to Project Natal for the XBox 360!  Now I'm almost certain this is the sensor referred to by Dieter Fox.

Recent Advances in Depth Cameras

A natural application for depth cameras is Simultaneous Localization and Mapping (SLAM).  Dieter Fox's work in building full 3D indoor maps using depth cameras is quite impressive.  The video below shows the map creation process with the "camera" view in the upper-left and the depth image in the lower-left.

A map fly-through conjures images of a Google Street View indoors.

Indeed, the resulting 3D map contains a lot of information!

A lot of researchers are ruminating that these sensors could revolutionize robotic perception, something akin to the impact of laser rangefinders -- after all, they essentially combine the functionality of cameras and laser rangefinders into one convenient device.  Perhaps the biggest advocates are convening at an upcoming Robotics Science and Systems (RSS) 2010 workshop entitled, "RGB-D: Advanced Reasoning with Depth Cameras."  Unfortunately I will probably not be in attendance; any chance the organizers could record the workshop and make it available on Google Video...?  (I would be willing to front a fraction of the registration costs for a live streaming video option...)

Brief Overview of Depth Camera Systems

Literally dozens of ranging / depth cameras have been developed.  Here are just a few of the ones I'm familiar with:

	The Mesa Imaging SwissRanger 4000 (SR4000) is probably the most well-known ToF depth camera.  It has a range of 5-8 meters, 176 x 144 pixel resolution over 43.6° x 34.6° field of view.  It operates at up to 54 fps, and costs about $9,000.  I've seen these used in a number of academic laboratories.
	The PMD Technologies CamCube 2.0 is a lesser-known, but equally impressive ToF depth camera.  It has a range of 7 meters, 204 x 204 pixel resolution with 40.0° x 40.0° field of view.  It operates at 25 fps, and last time I checked, it costs around $12,000.
	As part of the PR2 robot, Willow Garage has developed a dense stereo rig using visible projected light (embedded in the sensor head pictured above-left).  I refer to this class of depth camera system as "projected-light stereo."  In essence, a projector displays (pseudo) random patterns onto the scene to enable robust stereo feature extraction.  This method seems to produce very high-quality range data at framerate to produce point-clouds like the one shown below-left.  The Willow Garage system was (partially) explored in a paper at Humanoids 2009 by Rusu et. al.  The biggest drawback with the system in its current form is that it uses visible light.  I'm sure subsequent versions will transition to infrared. To my knowledge, these are not yet available for purchase.
	The sensor pictured is Microsoft's Project Natal, as announced at 2009's E3 convention.  A brief history...  Microsoft purchased ZCam from 3DV Systems in 2009.  ZCam had made some impressive ToF imagers in the past and had plans to make a custom-silicon solution for ranging webcams.  However, the safe bet is that Project Natal is actually infrared projected-light stereo, hence the three opening ports on the pictured device (two camera ports for stereo imaging, one extra port for IR projection). Presumably this device will be quite cheap; it is meant to be a mass-market gaming console sensor.  To my knowledge, these are not yet available for purchase.
	The PrimeSense depth camera is driven by a single-chip custom-silicon solution.  It projects a known infrared pattern onto the scene (perhaps a grid-like pattern?) and determines depth based on the pattern's deformation as captured by an infrared CMOS imager.  The resulting depth image is 640 x 480 pixels with a maximum throughput of 60fps.  An optional color CMOS imager returns 1600 x 1200 photos.  You can learn more here. To my knowledge, these are not yet available for purchase.

I have no experience with any of these depth cameras, but would love to hear others' impressions.  How good is the ranging resolution?  How about spot size (how well does each pixel represents a single light ray, or how close does it follow a pin-hole model)?  How does it compare to a laser rangefinder?  Please let me know your impressions in the comments.

The Dieter Fox Connection

As mentioned, currently-available depth cameras are quite expensive.  I am intrigued by Dieter Fox's recent statement about a sub-$100 depth camera that could hit stores later this year.  However, the whole thing is a bit perplexing.  Dr. Fox is the director of Intel Research (Seattle) and professor of Computer Science at University of Washington.  So, is Intel developing a depth camera (perhaps a custom silicon solution like 3DV Systems' ZCam or Primesense)?  Or is Prof. Fox's lab using a Microsoft product (Project Natal device)?  I'm quite intrigued; perhaps someone can clarify or elaborate.