The Need for Low Cost Sensors in Robotics -- Holiday Edition

Kinect Depth Camera (RGB-D Camera)

This is a guest essay by Fred Nikgohar.  Fred is the CEO of RoboDynamics, makers of the TiLR robot which was recently featured in a New York Times overview of telepresence robots.  Fred argues that we've reached a watershed moment in robotics facilitated by cheap 3D sensors like Microsoft's Kinect (ie. PrimeSense RGB-D camera) -- that the Kinect provides a roadmap where "the best solution to complex, low-cost sensing (or actuation for that matter) is to take advantage of affordable, mass-produced components, complementing them with the innovative use of software solutions that benefit from constantly declining prices of computation."

The holidays are upon us and once again the world goes shopping.

Santa Monica Place is a newly remodeled shopping mall directly across our offices here in beautiful downtown Santa Monica. Every holiday season we are witness to the large masses who visit the mall and the adjacent 3rd Street Promenade to do their holiday shopping. Given my distaste for crowded masses with Holiday Sales on their mind, I do my best to avoid shopping malls and stores at all costs during the holiday season - until this year.

There is one good reason for my excitement this year.

No doubt by now you've heard a great deal about the recently released Microsoft Kinect. Originally designed as a Game Controller for the XBox gaming platform, Kinect is a fantastic 3D sensor that retails for $150 in the U.S. What is for more interesting about the device however, is that it enables endless applications beyond just gaming. For example, if you've ever seen the movie Minority Report, you are already familiar with Gesture Recognition, the ability to interact/interface with digital content by simply gesturing to your computer. These types of interfaces are now a reality with the Kinect device.


Immediately following the $3,000 bounty by Adafruit for the first open source driver to read the data-stream from Kinect, we have been witness to a plethora of home-brewed projects from every corner of the world who are using the device in ways never imagined before. From gesture interfaces for controlling your computer to SLAM implementations on mobile robots... and everything in between (, Kinect has proven to be one heck of a sensor, and it is inspiring imagination and creativity all over the world.

But you know what's funny? 

Kinect is hardly the first - or even the most ideal sensor - to  enable these innovation. For at least a decade, we’ve had Time of Flight Cameras, RBGDs, LIDARs, Stereo Vision, and a number of other sensors that could have been used to create such applications. Yet somehow it was Kinect that kick-started such a flurry of innovation. Which naturally invites the question: what is it about Kinect that is inspiring such flurry of activity on a global scale?

In a single word: Affordability!

You see, all the sensors predating the Microsoft Kinect have been - and still remain - financially unaffordable to the average consumer. Ranging from several hundred to several thousands of dollars, they are inaccessible to consumers who clearly have the will, the skill, and the desire to create such amazing applications. The availability of Kinect at $150 retail changes the equation dramatically. It provides to the masses inexpensive access to a powerful sensor with a simple interface that can be used in virtually hundreds of applications - some as yet unimagined.

The speed at which these Kinect-based home-brewed innovations are proliferating around the Internet surprised even Microsoft, who originally threatened “legal action” against anyone who would attempt to hack it. In just under three weeks Microsoft did a complete reversal in attitude when it announced publicly that it had “left Kinect open by design” thereby encouraging the innovations we are now witness to. Regardless of Microsoft’s intentions, there is no denying the variety and quality of applications that are popping up everyday.

I’ve written before about the dire need for Low Cost Sensors in Robotics. Sensing is a huge challenge of commercializing consumer robotic platforms and applications. The inexpensive sensors are incapable of providing useful information about the robot’s environment, whereas the capable sensors are expensively cost-prohibitive. Though certain challenges like Obstacle Avoidance are manageable with inexpensive sensors, challenges like Autonomous Navigation remain woefully unresolved given the strict budgetary requirements of a consumer electronic product (such as a robot). And without capabilities like Autonomous Navigation, we find ourselves severely limited as to the type and usefulness of the robots we can produce at consumer-affordable prices.

The very existence of a complex yet (relatively) inexpensive sensor such as Kinect makes it possible for the first time to build (robotic) applications that are simultaneously useful and affordable. But how do we make complex sensors that are simultaneously capable and affordable? We can look to Kinect itself for the answer. In developing Kinect, Microsoft could have used its scale to drive down the price of exotic components. Instead, they chose to use mass-produced commodity components readily available in the market and achieve complexity by off-loading intelligence to a commodity CPU. The trade-off has a more costly upfront development cost, off-set through time as the product scales through market-acceptance, in return for a far cheaper piece of hardware made up of commodity components.  Inside a Kinect you will find two RGB cameras, a depth sensor, a microphone array, and a CPU to do the heavy lifting. These are all components produced in huge scale and readily affordable. And therein lies an important lesson for the robotics industry:

The best solution to complex, low-cost sensing (or actuation for that matter) is to take advantage of affordable, mass-produced components, complementing them with the innovative use of software solutions that benefit from constantly declining prices of computation.

There is always a paradox in disruptive technologies: new and exotic innovations of today become tomorrow’s commodity components, readily available in large quantity and high quality. The challenge is to drive market demand that leads to increased production, greater competition, and ultimately higher quality products and lower prices for the masses. CPU, GPS, WiFi, LCD Touch-screens, and countless other technologies all experience this trajectory.

3D sensing generally, and Kinect specifically, will also experience this same price/quality trajectory. By using inexpensive components and advance software algorithms, Microsoft has succeeded in introducing to the market an entirely new user interface paradigm. More importantly, they will succeed in creating the demand necessary for low cost 3D sensing, which benefits not only them as a company, but the entire digital technology space as a whole. It won’t be long before a new breed of 3D sensors hit the market, drastically decrease in price through time, and render gesture-based interactions commonplace in every home.

If we are to have a robot in every home, we must have low cost complex sensing. Kinect is showing us the roadmap. It is up to us to learn and apply the lessons... and as we do, we will come close to that day where man and machine will live together in harmony. My hope is that products such as Kinect, and the ingenuity of the home brew community, compels us to act. I look forward to your comments and questions.

Happy Holidays and Happy New Year.

Fred Nikgohar
CEO - RoboDynamics Corp. 



I would like to thank Fred for heeding Hizook's request for assistance and for submitting such a quality essay!   In the past, such essays were commonplace in SciFi magazines (eg. Marvin Minsky's "Telepresence: A Manifesto" in Omni Magazine).  I'm not sure that a common forum exists any longer, but with your discussion and submissions perhaps we can turn Hizook into a place for robotics thought leadership.



Enjoyed this discussion - I wonder in terms of the market sizing and how we will cause the market to grow.

Usually there is a purpose for growth - solving a particular problem.  All of these hacks are exciting in the fact that everyone has high-powered, low-cost processing today.  This state of development is partically due to the market growth and product adoption (e.g., the business community desires improved computer performance and/or Intel loves causing product obsolescence.

And with the Internet and our human desire for creation/sharing - we are able to make these things happen since all of us have moments of time where we can focus/donate to these creations.  How will a corporate entity manufacture these lower-cost devices without a driving need?  That I look forward to.

I see that Chris Anderson, founder of DIYDrones, made a blog post about this article.

The DIYdrones community can provide a really unique perspective about this issue: what role can / does the open hardware (hacker) community play in the development of personal (home) robots?  Is the early-adopter and hacker community (alone) able to provide sufficient economies of scale?

I'm already super encouraged by the success of DIYdrones in this regard -- they're proving that UAV enthusiasts are sufficient to support a decent-sized company (speaking of, I wonder if they hit their $1M projected revenue for 2010?).

Hopefully we can get some of the DIYdrones folks to lend us their perspective.

—Travis Deyle

Dear @Sanford,

My name is Fred Nikgohar [], author of this article.

Thank you for participating in this discussion... that is the most important thing I was hoping to come out of writing this.

There is no doubt in my mind that robotics will remain within the realm of hobby if we can't figure out compelling needs in the market. There are already a few identified market needs in existence, such as Unmanned Ground and Aerial vehicles, dynamic warehouse maintenance, precision surgical systems, research & education, and of course what we do at RoboDynamics which we call (Enterprise) Robotic Telepresence []. Despite the nascent nature of these fields, we are already seeing specialization and market segmentation in the market. It is not a surprise and closely tracks other technologies like PCs, Smartphones, WiFi, etc. Over time we will find other uses for these technologies and as more players enter the field, further competition will cause more specialization and segmentation.

The one thing I want to highlight, something that we have experienced first hand at RoboDynamics, is the degree to which a luxury becomes a "need" with declining prices. Customers who viewed our technology as a "nice-to-have" in 2003 ($35k pricepoint) now suddenly have a "need" for Robotic Telepresence at $10k. The point I'm making is that market needs must be defined far broader and more nuanced than just "solving a problem".

I look forward to your feedback.



From DIYDrones comments (with permission):

Thanks, Travis. I think we can provide some economies of scale, but not nearly what Microsft/China/consumer electronics/phones can. I think our business is always going to be measured in thousands or maybe tens of thousands of units, not hundreds of thousands or millions. By the time autopilots are out then in those numbers, they'll be toys, not hacker gear.

That said, the economies of scale at thousands are not to be sniffed at, and as your post points out we leverage the consumer electronics economics by using the same sensors that you can find in phones and game consoles, so thing get cheaper without us having to do anything.

As for our own business, we didn't quite hit $1m in 2010 (came a couple hundred k short) but will blow past it in 2011. We're running it at break-even now, rather than seeking to make a profit, because we're reinvesting any earnings into R&D for the benefit of the community, helping introduce better, cheaper, easier to use technology.

-- Chris Anderson


—Travis Deyle

I enjoyed reading this article and fully agree that we roboticists, whether hobbyist, academic, or commercial, need a diverse collection of low-cost sensors. This actually applies to the DIY / homebrew community as a whole.

Repurposing or hacking already existing but inexpensive hardware is generally a good path. But in reality there can be a huge mismatch between the needs of a particular robot and the capabilities provided by the sensor. This mismatch can make engineering a solution difficult or impossible. Let me provide an example I know from vision:

 Typically mass-produced image sensors are available from VGA (640x480) resolution and upwards through the megapixels nowadays. However the frame rates are generally limited to a few frames per second at high resolution and video at a lower resolution. This is understandable- these image sensors were developed for cellphone cameras or inexpensive digital cameras. However this set of specs (resolution and speed) are mismatched to what robots would need for many autonomous navigation tasks- We have found that a lower resolution combined with a higher frame yields better sensing, and in fact it is better to use less pixels when possible (otherwise you overload the processor). For example, for controlling the yaw angle of a 7" toy helicopter, we've performed this with as few as 8 (eight) pixels at a 130Hz frame rate, and we can make that same helicopter hover in place in a room using just 512 pixels total.

However to achieve these specs we had to design our own image sensor. This may sound complicated, until you realize that by just using hundreds or thousands of pixels rather than millions, you can use lower resolution (e.g. cheaper) processes, and can reduce the design time from person-years to person-months or even weeks. You can then have such designs prototyped for as little as $6k (doable for a small entrepreneur) and, if it works, have it mass produced in the thousands for dollars or a few tens of dollars apiece. Furthermore since we designed the image sensor we could select the interface to work with any processor (or microcontroller) we want rather than be at the mercy of some choking standard.

If we had stuck with using off-the-shelf megapixel image sensors, then we would have had to use DSPs and FPGAs to perform the same, which would have drawn too much power and weighed too much to be useful.

Remember that "more" is not necessarily "better", and can often be much worse!

This basic approach- drastically reducing one particular specification of a sensor design enough so that it can be inexpensively prototyped and manufactured using older processes- is something that probably could be applied to Kinect-like sensors. It probably could be applied to a wide variety of other sensors, and fill the need for cheap sensors addressed in your article.