Executing quadrotor trajectories accurately and therefore safely is a challenging task. State-of-the-art adaptive controllers achieve impressive trajectory tracking results with slight performance degradation in varying winds or payloads, but at the cost of computational complexity. Requiring additional embedded computers onboard, adding weight and requiring power. Given the limited computational resources onboard, a trade-off between accuracy and complexity must be considered. To this end, we implement "Neural-Fly" a lightweight adaptive neural controller to adapt to propeller damage, a common occurrence in real-world flight. The adaptive neural architecture consists of two components: (I) offline learning of a condition invariant representation of the aerodynamic forces through Deep Neural Networks (II) fast online adaptation to the current propeller condition using a composite adaptation law. We deploy this flight controller fully onboard the flight controller of the Parrot Bebop 1,showcasing its computational efficiency. The adaptive neural controller improves tracking performance by ≈60% over the nonlinear baseline, with minimal performance degradation of just ≈20% with increasing propeller damage.