ELEANOR (Energy- and Latency-Efficient Insertion of Objects using a robotic Arm equipped with an event-based camera and NeurOmoRphic hardware) builds on the results of the previous INRC3 project, in which a robotic arm learned to insert objects using force feedback alone. Now the project combines event-based image processing with an overarching neural state machine to optimize the positioning of objects in real time. In this case, the robotic arm should be able to insert a connector into a socket by identifying the correct socket using visual feedback. The project addresses key challenges of adaptive, neuromorphic automation. In collaboration with Intel, the fortiss research team has developed a neuromorphic algorithm. Under the leadership of Dr. Evan Eames from the Neuromorphic Computing competence field, control strategies were tested and optimized before being used in the real robot. The project was funded by the Bavarian State Ministry of Economic Affairs, Regional Development and Energy (StMWi).
ELEANOR (Energy- and Latency-Efficient Insertion of Objects using a robotic Arm equipped with an event-based camera and NeurOmoRphic hardware) builds on the results of the previous INRC3 project, in which a robotic arm learned to insert objects using force feedback alone. Now the project combines event-based image processing with an overarching neural state machine to optimize the positioning of objects in real time. In this case, the robotic arm should be able to insert a connector into a socket by identifying the correct socket using visual feedback. The project addresses key challenges of adaptive, neuromorphic automation. In collaboration with Intel, the fortiss research team has developed a neuromorphic algorithm. Under the leadership of Dr. Evan Eames from the Neuromorphic Computing competence field, control strategies were tested and optimized before being used in the real robot. The project was funded by the Bavarian State Ministry of Economic Affairs, Regional Development and Energy (StMWi).
Biologically inspired precision and intelligent image processing
The ELEANOR project is advancing robot precision by combining event-based image processing with neuromorphic control. While event-based cameras are widely used in mobile robotics research, ELEANOR takes a novel approach by mounting a camera on a robot arm. Attaching the camera enables real-time visual processing without the need to constantly move the arm and uses biologically inspired attention mechanisms to focus on relevant objects.
A key development of the project is the combination of specialized image processing algorithms with an adaptive controller. A neuromorphic pre-processing step intelligently segments the field of view, while a biologically inspired Dynamic Neural Field (DNF) focuses and tracks objects such as plugs. In addition, a DNF-based memory system enables the robot to store information about previously identified sockets, significantly increasing efficiency in complex tasks.
At the heart of ELEANOR's architecture is the integration of the aforementioned components together with the visual servo control into an overarching structure - the Neural State Machine. This is a decision-making system that replaces traditional logic gates with a fully neural structure that mimics human cognitive processes. For the first time, the entire processing pipeline, from perception to decision making, has been implemented using spiking neural networks.
Each component has been successfully tested on Intel’s neuromorphic research chip Loihi 2, demonstrating that neuromorphic computing can deliver the required performance for industrial applications while maintaining low power consumption and minimal latency. ELEANOR integrates event-based vision processing with neuromorphic control in a fully spiking system, marking a major step toward efficient and adaptive robotic automation. This advancement opens new possibilities, particularly for small and medium-sized enterprises (SMEs), to enhance the precision and efficiency of their operations.
Neuromorphic robotics unlocks new possibilities for adaptive automation
The knowledge gained and technologies developed in the ELEANOR project offer considerable added value, particularly for companies in the robotics and image processing sectors. With the expansion of neuromorphic capabilities, these innovations will become standard functions in numerous industrial sectors and thus form a basis for the next generation of intelligent automation. By using these advanced AI technologies, companies can increase their competitiveness and implement innovative solutions for complex automation tasks.
The ELEANOR project made an important contribution to the further development of neuromorphic robot control. The successful combination of event-based sensor technology with spiking neural networks on neuromorphic hardware demonstrates the potential of these technologies for precise and energy-efficient robot applications. Future applications can go beyond object insertion. Thus, the possibilities of robot control can be transferred to other tasks, such as gripping, manipulating or placing objects in situations where robot precision is required.
