Low-Cost Depth Cameras (aka Ranging Cameras or RGB-D Cameras) to Emerge in 2010?

Time-of-Flight (TOF) Ranging Cameras such as the SR4000 and PMDtec CamCube

Depth cameras go by many names: ranging camera, flash lidar, time-of-flight (ToF) camera, and RGB-D camera.  The underlying sensing mechanisms are equally varied: range-gated ToF, RF-modulated ToF, pulsed-light ToF, and projected-light stereo.  The commonality is that all provide traditional (sometimes color) images and depth information for each pixel (depth images) at framerate.  Existing commercial offerings, such as the Swiss Ranger SR4000 and PMD Tech products, currently cost ~$10,000.   Thus, I'm extremely excited by Dieter Fox's recent statement about a sub-$100 depth camera that could hit stores later this year!  Dr. Fox has already leveraged a similar (this?) sensor to build cool 3D SLAM maps akin to Google Street View indoors -- see videos below.  Is Dr. Fox's employer (Intel) building depth cameras?  Is this a new PrimeSense offering?  Or could it hail from fellow Seattle powerhouse, Microsoft, who not long ago purchased 3DV Systems (purveyor of ToF cameras) and who plans to release Project Natal (rumored to be projected stereo) later this year for the XBox 360?  I'd love details, but am intrigued regardless!  Updated March 31st 2010:  Big news -- PrimeSense is supplying the 3D sensing technology to Project Natal for the XBox 360!  Now I'm almost certain this is the sensor referred to by Dieter Fox.

 

Recent Advances in Depth Cameras

 

A natural application for depth cameras is Simultaneous Localization and Mapping (SLAM).  Dieter Fox's work in building full 3D indoor maps using depth cameras is quite impressive.  The video below shows the map creation process with the "camera" view in the upper-left and the depth image in the lower-left.

 

A map fly-through conjures images of a Google Street View indoors.

 

Indeed, the resulting 3D map contains a lot of information!

Depth Camera Map Indoors

 

A lot of researchers are ruminating that these sensors could revolutionize robotic perception, something akin to the impact of laser rangefinders -- after all, they essentially combine the functionality of cameras and laser rangefinders into one convenient device.  Perhaps the biggest advocates are convening at an upcoming Robotics Science and Systems (RSS) 2010 workshop entitled, "RGB-D: Advanced Reasoning with Depth Cameras."  Unfortunately I will probably not be in attendance; any chance the organizers could record the workshop and make it available on Google Video...?  (I would be willing to front a fraction of the registration costs for a live streaming video option...)

 

Brief Overview of Depth Camera Systems

 

Literally dozens of ranging / depth cameras have been developed.  Here are just a few of the ones I'm familiar with:

 SwissRanger SR4000 Ranging Camera The Mesa Imaging SwissRanger 4000 (SR4000) is probably the most well-known ToF depth camera.  It has a range of 5-8 meters, 176 x 144 pixel resolution over 43.6° x 34.6° field of view.  It operates at up to 54 fps, and costs about $9,000.  I've seen these used in a number of academic laboratories.
 CamCube2.0 Depth Camera from PMD Technologies The PMD Technologies CamCube 2.0 is a lesser-known, but equally impressive ToF depth camera.  It has a range of 7 meters, 204 x 204 pixel resolution with 40.0° x 40.0° field of view.  It operates at 25 fps, and last time I checked, it costs around $12,000.

Projected (Active) Stereo by Willow Garage's PR2 Robot

Projected (Active) Stereo by Willow Garage's PR2 robot

As part of the PR2 robot, Willow Garage has developed a dense stereo rig using visible projected light (embedded in the sensor head pictured above-left).  I refer to this class of depth camera system as "projected-light stereo."  In essence, a projector displays (pseudo) random patterns onto the scene to enable robust stereo feature extraction.  This method seems to produce very high-quality range data at framerate to produce point-clouds like the one shown below-left.  The Willow Garage system was (partially) explored in a paper at Humanoids 2009 by Rusu et. al.  The biggest drawback with the system in its current form is that it uses visible light.  I'm sure subsequent versions will transition to infrared.

To my knowledge, these are not yet available for purchase.

 Project Natal from Microsoft

The sensor pictured is Microsoft's Project Natal, as announced at 2009's E3 convention.  A brief history...  Microsoft purchased ZCam from 3DV Systems in 2009.  ZCam had made some impressive ToF imagers in the past and had plans to make a custom-silicon solution for ranging webcams.  However, the safe bet is that Project Natal is actually infrared projected-light stereo, hence the three opening ports on the pictured device (two camera ports for stereo imaging, one extra port for IR projection). Presumably this device will be quite cheap; it is meant to be a mass-market gaming console sensor. 

To my knowledge, these are not yet available for purchase.

 PrimeSense Depth Camera

The PrimeSense depth camera is driven by a single-chip custom-silicon solution.  It projects a known infrared pattern onto the scene (perhaps a grid-like pattern?) and determines depth based on the pattern's deformation as captured by an infrared CMOS imager.  The resulting depth image is 640 x 480 pixels with a maximum throughput of 60fps.  An optional color CMOS imager returns 1600 x 1200 photos.  You can learn more here.

To my knowledge, these are not yet available for purchase.

 

I have no experience with any of these depth cameras, but would love to hear others' impressions.  How good is the ranging resolution?  How about spot size (how well does each pixel represents a single light ray, or how close does it follow a pin-hole model)?  How does it compare to a laser rangefinder?  Please let me know your impressions in the comments.

 

 

The Dieter Fox Connection

 

As mentioned, currently-available depth cameras are quite expensive.  I am intrigued by Dieter Fox's recent statement about a sub-$100 depth camera that could hit stores later this year.  However, the whole thing is a bit perplexing.  Dr. Fox is the director of Intel Research (Seattle) and professor of Computer Science at University of Washington.  So, is Intel developing a depth camera (perhaps a custom silicon solution like 3DV Systems' ZCam or Primesense)?  Or is Prof. Fox's lab using a Microsoft product (Project Natal device)?  I'm quite intrigued; perhaps someone can clarify or elaborate.

 

Comments

PrimeSense claims that their chips will ship at $20-30 in volume
—Anonymous

@ Anonymous,

You're right.  I completely forgot to include PrimeSense -- they were on my list as a major player in this space!  Their reference design is very telling about how the system functions.  It appears to be a single-chip custom-silicon solution. 

PrimeSense Depth Camera Reference Design

Given the price-point you mentioned, perhaps this is the one alluded to by Dieter Fox?

—Travis Deyle
I'll also add that, to the best of my knowledge, PrimeSense is the hardware of Natal, not ZCam, which should hint at when it might be available.
—Anonymous

@ Anonymous,

Do you know if Natal uses the same IR projector + camera combo for distance sensing as PrimeSense (is Microsoft using PrimeSense's product?), or does it use stereo estimation like Willow Garage's system?

—Travis Deyle

It's disappointing that MS bought the 3DV ip but aren't going to use the technology- I guess it's dead and buried unless some other vendor releases something similar.  I'm really curious if some of the demos and videos aren't using the PrimeSense (because they hadn't decided on it yet)- some of the videos show really low-res ~64x64 point clouds where the PrimeSense ought to be 640x480. 

I think the hardware stereo view cameras are interesting- there is Tzyx's G3 which is all hardware (and also in the $10K range). I think there was another also but I can't remember the vendor's name.   

I would be curious of your opinion since you have more experience in this area...

But to further extend your comments: It seems an RGB-D camera such as the PrimeSense will significantly change the depth sensing market, expecially at this price point. Do we even need Sick, Hokuyo anymore? My understanding is that previously it was a trade off between refresh rate and sensor resolution for the two sensor types (e.g. Hokuyo vs. SwissRanger). But at 640x480 (even ignoring the price difference) the PrimSense package seems to offer twice the resolution of a SwissRanger, and is now a better choice than a Hokuyo type sensor. Am I missing some other capability? Furthermore, if FOV is the concern, several PrimeSense units could be arrayed together for a cost that is still an order of magnitude less than the current offerings. Is this the death of Sick, SwissRanger and other high end depth sensor manufacturers?

—Anonymous
To use the PrimeSense in an array together might not work at all. I assume that they wont project a pattern unique to one camera and therefore using more than one PrimeSense will confuse the system. Those expensive cameras like the CamCube work in groups. (At this price, they better do... ;-) )
—Anonymous
Thanks for commenting about the suitability of the PrimeSense for arrayed configuration - my comment was the previous one. Assuming the patterns they project do not over lap - so the camera's FOV is equivalent to the projector's FOV - I assume a camera projected into an adjacent point of space should not be interfered with even if the projections are the same pattern. The scenario I would be concerned with interference is when multiple robots in the same space are using the same pattern projection with overlapping FOV's. But my understanding is this is the same problem that plagues the Hokuyo and other similar scanning sensors. Hopefully they will allow the user to select from multiple patterns, or make it user configurable if they provide a product for the robotics community - since this would make it usable for multiple robot environments.
—Anonymous

Omek Interactive is another Israeli company recently in the press with technology similar to Primesense, and ambitions to sell to the same consumer markets. This is great since competition should bring down cost and improve the technology.

I just wonder where this leaves the high cost yet lower resolution alternatives also mentioned in this post? e.g. Swiss ranger, PMD, ROS's expensive videre rig

 

—Anonymous

Interesting. Maybe you will find this of also of interest:

http://www.youtube.com/watch?v=UGGKh3cyc7w

 

 

—Anonymous

Let the age of inexpensive framerate 3D sensing begin!  The world's first mass-market, low-cost depth camera (or RGB-D camera if you prefer) started shipping yesterday in the form of Microsoft's Kinect, powered by PrimeSense.  See this Hizook article and its associated comments for details.

Depth Camera (RGB-D Camera)

 

—Travis Deyle

Just two comments:

The Panasonic-3D-TOF cam is an alomust one-to-one copy of an early Swiss Ranger, I think II, but not sure about the number.

The Swiss Range, PMD-Tech like 3D-Cams are to some extend low volume next to Hand-Made cameras. The low volume together with very individual support makes the high prices.

If one of the Cameras is taylored to such a low end application as gaming is and purchasing and production subcontracting power of large companies as well as high volume comes into play, cameras of this type can be in the 200$ range as well.

You can buy 3D-ToF cameras at iee.lu, specially taylored for security and surveillance application, produced and developed to automotive standards, suitable for professional used at 1000 ... 500 Euro. It is similar to the SwissRanger.

—henryb_de

Thanks for the shout-out to Omek Interactive but an important point of clarification -- Omek is solely a middleware company.  We have developed a motion tracking and gesture recognition software solution that works with nearly all 3D depth cameras available on the market today.  Omek is unique in that we are not owned or tied to any one camera provider.  We are the only sensor-independent software provider.  It allows developers and OEMs to take advantage of the rapid changes that are happening in this industry by developing an application or solution once and being able to deploy it on any of the existing (or future) depth cameras.  

Our long-range solution, BeckonTM Development Suite, is available as a free non-commercial download here: http://www.omekinteractive.com/products/beckon-3-for-pc/

We also have an embedded option that works with the BeagleBoard-xM.  Finally, we will be releasing a Beta later this year for our GraspTM Development Suite which provides accurate close-range hand and finger gesture recognition and tracking.  You can sign up to be notified of the Beta here: http://www.omekinteractive.com/products/grasp/

Alona

Post new comment

The content of this field is kept private and will not be shown publicly.
  • Web page addresses and e-mail addresses turn into links automatically.
  • Allowed HTML tags: <a> <em> <strong> <cite> <code> <ul> <ol> <li> <dl> <dt> <dd> <p>
  • HTML tags will be transformed to conform to HTML standards.

More information about formatting options