EMMANÜELA (Energy-efficient human-machine interaction sensor via edge AI for adaptive AR/VR) focuses on the development of a new type of energy-efficient sensor for human-machine interaction in the field of augmented and virtual reality (AR/VR). A central aspect is the direct processing of sensor data in an extended reality device, which ensures low latency times and high sensor reactivity - without the need to communicate with external servers. To achieve this, neuromorphic hardware that imitates the way the human brain works and event-based cameras are used, which enable energy-efficient, low-latency processing directly on the device (edge AI) - in contrast to classic, energy-intensive AI-based image analyses.
The fortiss scientists from the Neuromorphic Computing competence field contribute their experience in action recognition and the development of neuromorphic algorithms. They generate and analyse event-based data sets - i.e., data triggered by specific events - for motion detection and implement the algorithms on neuromorphic hardware. The prototype, which is demonstrated in AR glasses, is also co-developed by fortiss.
Simi Reality Motion Systems GmbH provides its expertise in camera-based motion analysis systems and uses standard hardware to improve the transparency and efficiency of motion data. Simi's technologies contribute to the development of the sensor technologies and complement the neuromorphic approach of the project.
Optimized interaction between man and machine
The project uses event cameras that are able to capture events in real time, as well as inertial sensors that measure acceleration and rotation rates to enable precise motion detection in AR glasses. Event cameras register changes in the field of view with high temporal resolution, reducing the amount of data and minimizing delays compared to conventional cameras. Neuromorphic algorithms optimize this process and reduce energy consumption. The combination of neuromorphic hardware and event cameras makes it possible to bring high-performance, low-power and low-latency motion detection directly to the device.
This saves the system delaying communication with external servers. Reactive AR/VR dynamically adapts content to user actions and changes in the environment and uses sensors or cameras for interaction. Immersive AR/VR combines realistic graphics, 3D audio and haptic feedback to create the most immersive digital experience possible.
Research in the context of immersive digital worlds
EMMANÜELA ties in with the latest developments in the field of extended reality (XR), where digital and real experiences merge. Users interact through AR, VR and other technologies to experience a seamless and immersive experience. The research results contributed by fortiss could enable new, intuitive interaction methods for AR/VR that minimize the need for external hardware while increasing efficiency and user-friendliness.
Through the development of energy-efficient sensors and novel algorithms, the project promotes the continuous improvement of sensor technology and the reduction of energy consumption. These advances offer numerous innovative applications that can play a significant role beyond industry in medicine, education and entertainment.
