Insects like honeybees exhibit remarkable navigational abilities despite their simple nervous systems, showcasing expertise in tasks such as long-distance travel, landmark recognition, and spatial memory. These skills are crucial for efficient foraging and homing. In robotics, one of the main challenges is to navigate in GPS-denied environments with limited sensors and processors onboard. In this study, we propose a novel navigation strategy that uses learning flights during which a robot directly maps images to nest location vectors, inspired by honeybees. In our previous research, the learning phase was modeled using compact convolutional neural networks (CNNs) and demonstrated successful learning and control in simulation. In this work, we combine this homing model with odometry and implement both in a real-world quadrotor. More specifically, we examine how this model can compensate for the drone’s odometric drift to reach home after it performs a long-distance outbound flight. Real-world experiments demonstrate the proof of concept of the proposed navigation strategy in both indoor and outdoor flights.

Spiking neural networks implemented for sensing and control of robots have the potential to achieve lower latency and power consumption by processing information sparsely and asynchronously. They have been used on neuromorphic devices to estimate optical flow for micro air vehicles navigation, however robotic implementations have been limited to hardware setups with sensing and processing as separate systems. This article investigates a new approach for training a spiking neural network for optical flow to be deployed on the speck2e device from Synsense. The method takes into account the restrictions of the speck2e in terms of network architecture, neuron model, and number of synaptic operations and it involves training a recurrent neural network with ReLU activation functions, which is subsequently converted into a spiking network. A system of weight rescaling is applied after conversion, to ensure optimal information flow between the layers. Our study shows that it is possible to estimate optical flow with Integrate-and-Fire neurons. However, currently, the optical flow estimation performance is still hampered by the number of synaptic operations. As a result, the network presented in this work is able to estimate optical flow in a range of [-4, 1] pixel/s.

We conduct a simulation study of an insect-inspired navigation method that combines visual learning in a small area around a home location with path integration to successfully navigate over distances 8 to 10 times larger than the learning radius, while only requiring 6MB of memory. Our simulations show that the method is capable of learning with on-board data only and handling the noise levels and errors it may encounter in real-world conditions. Furthermore, we investigate the integration of online learning with a structured buffer to mitigate catastrophic forgetting, allowing continuous learning during flight. Additional experiments assess the system's generalization capabilities beyond the learned area, its robustness to small variations in pitch and roll, and the relationship between network size and the learnable area size.

Biological sensing and processing is asynchronous and sparse, leading to low-latency and energy-efficient perception and action. In robotics, neuromorphic hardware for event-based vision and spiking neural networks promises to exhibit similar characteristics. However, robotic implementations have been limited to basic tasks with low-dimensional sensory inputs and motor actions because of the restricted network size in current embedded neuromorphic processors and the difficulties of training spiking neural networks. Here, we present a fully neuromorphic vision-to- control pipeline for controlling a flying drone. Specifically, we trained a spiking neural network that accepts raw event-based camera data and outputs low-level control actions for performing autonomous vision-based flight. The vision part of the network, consisting of five layers and 28,800 neurons, maps incoming raw events to ego-motion estimates and was trained with self-supervised learning on real event data. The control part consists of a single decoding layer and was learned with an evolutionary algorithm in a drone simulator. Robotic experiments show a successful sim-to- real transfer of the fully learned neuromorphic pipeline. The drone could accurately control its ego-motion, allowing for hovering, landing, and maneuvering sideways—even while yawing at the same time. The neuromorphic pipeline runs on board on Intel’s Loihi neuromorphic processor with an execution frequency of 200 hertz, consuming 0.94 watt of idle power and a mere additional 7 to 12 milliwatts when running the network. These results illustrate the potential of neuromorphic sensing and processing for enabling insect-sized intelligent robots.@en

Insects have long been recognized for their ability to navigate and return home using visual cues from their nest's environment. However, the precise mechanism underlying this remarkable homing skill remains a subject of ongoing investigation. Drawing inspiration from the learning flights of honey bees and wasps, we propose a robot navigation method that directly learns the home vector direction from visual percepts during a learning flight in the vicinity of the nest. After learning, the robot will travel away from the nest, come back by means of odometry, and eliminate the resultant drift by inferring the home vector orientation from the currently experienced view. Using a compact convolutional neural network, we demonstrate successful learning in both simulated and real forest environments, as well as successful homing control of a simulated quadrotor. The average errors of the inferred home vectors in general stay well below the 90° required for successful homing, and below 24° if all images contain sufficient texture and illumination. Moreover, we show that the trajectory followed during the initial learning flight has a pronounced impact on the network's performance. A higher density of sample points in proximity to the nest results in a more consistent return. Code and data are available at https://mavlab.tudelft.nl/learning_to_home.

@en

Neuromorphic sensors, like for example event cameras, detect incremental changes in the sensed quantity and communicate these via a stream of events. Desired properties of these signals such as high temporal resolution and asynchrony are not always fully exploited by algorithms that process these signals. Spiking neural networks (SNNs) have emerged as the algorithms that promise to maximally attain these characteristics and are likely the key to achieving a fully neuromorphic computing pipeline. But, this means that if the SNN is to take full advantage, the event stream must be sent directly and unaltered to the SNN, which in turn implies that all temporal integration should occur inside the SNN. Therefore, it is interesting to investigate the mechanisms that achieve this. This thesis does so through evaluating and comparing the performance of different memory mechanisms in SNNs found in the literature, as well as through an in depth analysis of the inner workings of these mechanisms. The mechanisms include spiking neural dynamics (leaks and thresholds), explicit recurrent connections, and propagation delays. We demonstrate our concepts on two small scale generated 1D moving pixel tasks in preliminary experiments first. After that, we extend our research to compare the memory mechanisms on a real-world neuromorphic vision processing task, in which the networks regress angular velocity given event based input. We find that both explicit recurrency and delays improve the prediction accuracy of the SNN, compared to having just spiking neuronal dynamics. Analysis of the inner workings of the networks shows that the threshold and reset mechanism of spiking neurons play an important role in allowing longer neuron timescales (lower membrane leak). Forgetting (at the right time) turns out to play an important role in memory. Additionally, it becomes apparent that optimizing an SNN with explicit recurrent connections or learnable delays does not lead to the formation of robust spiking neuronal dynamics. In fact, spiking neuronal dynamics are largely ignored, as after optimization virtually no input current is integrated onto the membrane potential in these cases. Instead, we consistently find that a recurrent SNN prefers to build a state solely with the explicit recurrent connections, while an SNN with delays prefers to just use the delays. Therefore, our SNNs with explicit recurrent connections and delays are in fact better described as binary activated RNNs and ANNs, respectively.

Neuromorphic processors like Loihi offer a promising alternative to conventional computing modules for endowing constrained systems like micro air vehicles (MAVs) with robust, efficient and autonomous skills such as take-off and landing, obstacle avoidance, and pursuit. However, a major challenge for using such processors on robotic platforms is the reality gap between simulation and the real world. In this study, we present for the very first time a fully embedded application of the Loihi neuromorphic chip prototype in a flying robot. A spiking neural network (SNN) was evolved to compute the thrust command based on the divergence of the ventral optic flow field to perform autonomous landing. Evolution was performed in a Python-based simulator using the PySNN library. The resulting network architecture consists of only 35 neurons distributed among 3 layers. Quantitative analysis between simulation and Loihi reveals a root-mean-square error of the thrust setpoint as low as 0.005 g, along with a 99.8% matching of the spike sequences in the hidden layer, and 99.7% in the output layer. The proposed approach successfully bridges the reality gap, offering important insights for future neuromorphic applications in robotics. Supplementary material is available at https://mavlab.tudelft.nl/loihi/.@en

Flying insects are capable of vision-based navigation in cluttered environments, reliably avoiding obstacles through fast and agile maneuvers, while being very efficient in the processing of visual stimuli. Meanwhile, autonomous micro air vehicles still lag far behind their biological counterparts, displaying inferior performance at a much higher energy consumption. In light of this, we want to mimic flying insects in terms of their processing capabilities, and consequently show the efficiency of this approach in the real world. This letter does so through evolving spiking neural networks for controlling landings of micro air vehicles using optical flow divergence from a downward-looking camera. We demonstrate that the resulting neuromorphic controllers transfer robustly from a highly abstracted simulation to the real world, performing fast and safe landings while keeping network spike rate minimal. Furthermore, we provide insight into the resources required for successfully solving the problem of divergence-based landing, showing that high-resolution control can be learned with only a single spiking neuron. To the best of our knowledge, this work is the first to integrate spiking neural networks in the control loop of a real-world flying robot. Videos of the experiments can be found at https://bit.ly/neuro-controller.

@en