WaveTune: Harnessing Radar Sensors to Compose Music Beats with Body Gestures

Abstract

In the field of music technology, the ability to connect body movements with music generation has led to highly engaging applications. Many existing solutions use cameras or wearable devices to capture physical movements and translate them into commands for a music generation tool. While the camera-based systems are privacy-invasive and demand good lighting conditions, the wearable devices adversely impact the fluidity of movements. We propose WaveTune, a privacy-friendly and device-free interface for creating music beats through body gestures. Wavetune has the following components: (i) a millimeter-wave radar as the frontend sensor that captures body movements, (ii) a gesture recognition algorithm that performs gesture delimitation and identification in real-time, and (iii) a music generation component that maintains a seamless and pleasant experience. WaveTune also provides an option to continuously map random dynamic movements into musical parameters to encourage further interaction. We recruited 24 users to train and test WaveTune’s ability to map body gestures to musical commands. To the best of our knowledge, WaveTune is the first mmWave system that allows a layered composition of music beats.