Unreal Success: Vision-Based UAV Fault Detection and Diagnosis Framework

Abstract

Online fault detection and diagnosis (FDD) enables Unmanned Aerial
Vehicles (UAVs) to take informed decisions upon actuator failure during
flight, adapting their control strategy or deploying emergency systems.
Despite the camera being a ubiquitous sensor on-board of most commercial
UAVs, it has not been used within FDD systems before, mainly due to the
nonexistence of UAV multi-sensor datasets that include actuator failure
scenarios. This paper presents a knowledge-based FDD framework based on
a lightweight LSTM network and a single layer neural network classifier
that fuses camera and Inertial Measurement Unit (IMU) information.
Camera data are pre-processed by first computing its optical flow with
RAFT-S, a state-of-the-art deep learning model, and then extracting
features with the backbone of MobileNetV3-S. Short-Time Fourier
Transform is applied on the IMU data for obtaining their time-frequency
information. For training and assessing the proposed framework, UUFOSim
was developed: an Unreal Engine-based simulator built on AirSim that
allows the collection of high-fidelity photo-realistic camera and sensor
information, and the injection of actuator failures during flight. Data
were collected in simulation for the Bebop 2 UAV with 16 failure cases.
Results demonstrate the added value of the camera and the complementary
nature of both sensors with failure detection and diagnosis accuracies
of 99.98% and 98.86%, respectively.