Eye-mimicking AI car camera detects pedestrians, obstacles 100x faster

While talking to Interesting Engineering, Dr Scaramuzza, Director of the Robotics and Perception Group, UZH, explained that by combining a bio-inspired event camera with artificial intelligence (AI), the new system addresses the limitations of current technology, offering faster detection while using less computational power.

The challenge of obstacle detection

Traditional frame-based cameras used in driver assistance systems capture snapshots at regular intervals, typically 30 to 50 frames per second.

“But if something happens during the 20 or 30 milliseconds between two snapshots, the camera may see it too late,” said Daniel Gehrig, the first author of the paper.

“The solution would be increasing the frame rate,” commented Gehrig.

However, increasing the frame rate would require more computational power and data processing, posing a challenge for real-time applications.

A magical hybrid solution

To overcome these limitations, researchers combined the event camera with a standard camera capturing 20 images per second. The standard camera’s images are processed by a convolutional neural network trained to recognize cars and pedestrians, while the event camera’s data is analyzed by an asynchronous graph neural network, specializing in 3D data that changes over time.

This hybrid system leverages detections from the event camera to anticipate and enhance detections by the standard camera.

Remarkably, the system can effectively detect objects that enter the field of view between frames of the standard camera, significantly improving safety, especially at high speeds.

“I used to work part-time as a magician,” Scaramuzza told IE, “and I still do perform magic tricks for family and friends. In general, I like things that seem at first glance impossible. Like Nelson Mandela used to say: ‘Everything seems impossible until it’s done.'”

Faster detection, wide adaptability

Scaramuzza explained that while the system hasn’t been tested on a car in closed-loop control, it has been rigorously evaluated on both public and newly recorded datasets.

“Performance was evaluated against the go-to state of the art system showing that we can achieve 100 times faster detections at less than 1/100th of the bandwidth.”

When asked about the datasets used in the study, Scaramuzza revealed, “We recorded and released a new dataset called DSEC DETECTIONS which is now public.”

The researcher also shed light on the scalability of the hybrid system. He confirmed that while their demonstration focused on a front-facing stereo camera, their technology can be “readily extended to multiple cameras.”

Moreover, the system has not faced any obstacle in fusing data from multiple cameras in real time, added Scaramuzza.

Future implications seem impressive

The researchers envision that their method could be further enhanced by integrating cameras with LiDAR sensors, commonly used in self-driving cars.

“Hybrid systems like this could be crucial to allow autonomous driving, guaranteeing safety without leading to a substantial growth of data and computational power,” added Scaramuzza.

This breakthrough technology has the potential to transform driver assistance systems, making them faster, more reliable, and safer. As autonomous vehicles become a reality, such advanced obstacle detection systems will be essential for ensuring the safety of both drivers and pedestrians.