Diversity matters: The effect of subject and environmental variables in test data on markerless motion capture accuracy

Abstract

3D human motion capture can provide unique information on joint angles and position, and body segment lengths. It has many potential medical applications, including diagnostics, disease monitoring, and clinical decision making. Markerless motion capture makes motion capture accessible and has the potential to objectify motion analysis, and possibly reduce bias. However, neural networks for markerless motion capture can become biased by using unrepresentative training data. Mainstream datasets used for training do not provide detailed information about their subjects characteristics, like skin tone and clothing types per video, making it difficult to study their diversity. This study uses the OpenSim Driven Animated Human (ODAH) pipeline to create three datasets with increasingly diverse subjects and environments and assess the bias and generalization of a biomechanics-aware neural network. The Mean Per Joint Absolute Error (MPJAE), Procrustes Aligned Mean Per Joint Position Error (PA-MPJPE), and absolute relative scale error are determined per video and statistically analyzed together with the video's variables. Significant differences were found in the MPJAE and PA-MPJPE for videos with different motion types, subjects, skin tones, and blinds colors, and significant correlations were found with weight and BMI. The scale error differed significantly based on the subject, skin tone, environmental colors, weight, and height. The model performed significantly better on seen skin tones and environments than unseen skin tones and variables. This study has shown that it cannot be assumed that a neural network’s performance will generalize to different skin tones and different environmental colors, because this will result in greater errors. In future research, the created test data should be expanded to retrain the network and determine the effect of increasing training data diversity.