“Light,” a former camera company has announced a new depth sensor that could be a game changer, upending LIDAR and computer vision based depth measurement, by producing a combination RGB image and depth map with ranges out to an astonishing 1000 meters. Presuming it works as promised and can be delivered at scale, this is astonishing news.
Some of you will know Light as a company that made a computational camera with 16 lenses at 3 different focal lengths. Combining the images let them produce a high resolution image with an adjustable depth of field, hoping to get SLR quality in a flat box. The tested their first product, and while interesting, it was not ready for prime-time. They later got money from Softbank. My presumption was they would attempt to use computational photography to estimate distance to pixels, but they’ve done that beyond any expectation I had, since 1000m of range did not seem possible.
My guess is that the light Clarity takes camera images with several cameras focused at different distances. It also uses a fairly long baseline stereo. As photographers know, an image from a fast lens will be sharp at the focal distance, and more blurry the farther away something is from that distance. The range over which the image is sufficiently sharp for your sensor is the “depth of field.” With two or more images at different focal distances, you can compare how blurry the images are and estimate the actual distance — but that’s not enough to get high accuracy out to 1000m, nor is long baseline stereo, so they must have gone beyond this basic technique.
While they have not announced prices, such a system should not be expensive. Sensors and lenses are cheap, and solid state. Computation gets cheaper with time and custom silicon. This differs from most LIDARS which use more expensive components and often have moving parts or parts that must be calibrated to extreme precision, making it difficult to keep them aligned inside the high vibration environment of a car. While extensive effort has been made to solve these problems with LIDARs, work still continues. The Clarity claims to offer everything the LIDAR does (with one big exception) and to do it over far greater range. I have not yet seen details on how the range values will compare for accuracy and precision with LIDAR.
The “big exception” is that LIDAR uses emitted light, so it works the same regardless of illumination, day and night. The LIDAR sees an object in the black of night, the Clarity will need it to be illuminated by headlights or other sources, the same way human eyes and cameras do. However, being as good as a human is a pretty reasonable performance.
Another exception is that some LIDARs deliver the speed of a target, like radar does. This remains valuable. On the other hand, with higher frame rates and enough accuracy and precision, you can measure the speed of objects fairly well.
The other competition is the use off computer vision, sometimes called pseudo LIDAR. This is analogous to how humans figure the depth to things, using our brains and knowledge of the world. This, and other computer vision techniques not only suffer from needing illumination, but have to deal with changing lighting and shadows which alter how things look. This should not greatly affect the Clarity technique.
The Clarity outputs a distance point cloud, pretty much the same as LIDARs do. It should be usable as a drop-in replacement for directional LIDAR. (Some LIDARs offer a 360 degree view, which the Clarity does not do unless you install several.) Light claims a speed of 30 frames per second, which is better than most LIDARs.
In addition, the Clarity offers the ordinary camera photographic image perfectly calibrated with its depth map. This is highly useful — everybody wants to match their LIDAR point clouds with their camera images to fuse the sensors, but this technique comes pre-fused. Neural networks can now classify objects from a full 3-D photograph of them, which should improve their performance.
A thousand meter range is astonishing. It’s for more than most automotive radars. Most 905nm LIDARs only see reliably around 120m while they can see some things out a bit further. This has led many companies to work with 1550nm LIDARs which can reliably see to 250m. You need to see 250m to drive fast on the freeway, especially in wet conditions, if you want to stop in time. It’s always good to see better.
Radar sees fairly far and of course works day and night. It also sees through fog and dust, and even detects objects you can’t see by bouncing off the road, letting you track the speed of vehicles blocked from view by the truck you’re following. It will remain in the sensor portfolio. It also gives you the speed.
TSLA famous “LIDAR is a crutch”
As a drop-in replacement for LIDAR, the Clarity, if it delivers, would kill much of the LIDAR industry but validate all those who have built to use it. Elon Musk famously does not use LIDAR, feeling that one can and should solve all problems with computer vision. It’s pretty hard to imagine why you wouldn’t want to use this, however, since you can combine the best of both worlds. Fortunately for Tesla, it might be a pretty minimal modification to Tesla vehicles, even existing ones, but it will be more of a software retrofit. Superhuman vision that also sees distance would not be a crutch, it would be an improvement on both LIDAR and vision approaches.
What’s not to love?
There’s very little not to love, though there are things we don’t know. I have reached out to Light to get answers on some of these questions.
- What will it end up costing? At present, anything under $1,000 would be a no-brainer for most teams, but it seems it could be much less. It needs to be less if you want 360 degree vision.
- When will it be available in large quantities for deployment?
- What is the precision? In their demo video they identify points just 4cm apart. Many LIDARS can do better than that, but in many cases it is not strictly needed. One key application — spotting potholes, bumps and debris on the road — likes good precision, but identifying a pedestrian does not.
- How well will it work at night under headlights?
- What is the resolution of the depth maps? The images shown suggest the depth map is much lower resolution than the RGB image, but still better than LIDAR.
- How accurate are the results? Low accuracy will also inherently reduce precision, but otherwise some error in this can be tolerated.
- It does seem to produce some errors with objects moving in front of other objects. We’ll need to learn the scope of these errors and if they affect any key detection algorithms.
- Will it be as robust as I suggest? This may be mountable by the rear-view mirror location, which is fantastic for many reasons, including the ability of the wipers to clear the view in rain.