Power-efficient gesture and movement recognition with neuromorphic hardware
Humans intuitively communicate information via gestures. By using the same paradigm for interacting with technical devices, human-machine interfaces can be enhanced. Requirements for efficient gesture interfaces are high accuracy, low response time, low power consumption and adaptivity. Neuromorphic technologies allow to optimize gesture recognition systems according to these metrics.
The proposed hardware setup consists of an event-based vision sensor and a neuromorphic processor. Event-based vision sensors detect changes in luminoscity in each of its pixels asynchronously. This leads to high temporal resolution, low power consumption, high dynamic range and data sparsity. Moreover the sensed events can be directly streamed into a neuromorphic processor and processed as spikes in a Spiking Neural Network (SNN). This new computing paradigm keeps up the low latency and low power consumption requirements. Additionally neuromorphic processors like Intel research chip Loihi allow for online, on-chip adaptation. This can be leveraged to teach new gestures to the network on-the-fly.
This project builds up upon recent works by Prof. Neftci who proposed a two-staged approach to implement Spiking Neural Networks on neuromorphic hardware. This project will contribute to the state-of-the-art in both stages.
The first stage is offline pretraining of the SNNs with a pre-collected dataset. New SNN architectures, SNN training algorithms and specific pretraining strategies (e.g. Meta-Learning) will be explored. These techniques optimize the SNN for the second stage, the online, on-chip learning. With the availability of a new generation of Intel‘s Loihi chip (Loihi 2), advancements to biologically-inspired learning rules will be enabled. This allows for efficient adaption of gestures and introduction of new gestures during system runtime.
All these ideas are implemented in a prototype which enables real-time gesture recognition and learning and makes neuromorphic technologies perceivable for everyone.
Bayerische Forschungsstiftung
Project number: AZ-1558-22
01.04.2023 - 30.09.2024
