Prediction Models for Individuals' Control Skill Development and Retention using XGBoost and SHAP

Abstract

Current capabilities for predicting skill retention, i.e., the extent to which human operators retain learned skills over time, at an individual level are limited due to a requirement for large data sets and methods that can extract relevant patterns in highly dimensional data. This paper investigates the application of Extreme Gradient Boosting (XGBoost) decision tree models for predicting a high-resolution individual skill retention curve. For this, a large skill-based tracking experiment dataset is used to extract different feature classes and train an XGBoost predictive model. To identify the robust predictors, the effects of the different features on the model's output are analyzed using SHapley Additive exPlanations (SHAP). Furthermore, the proposed XGBoost model is trained using both the experiment dataset and a matched synthetic dataset, with both approaches evaluated on the experiment data. Overall, the available experiment dataset was found to include too few retention measurements, and too significant between-group differences, to extract a reliable prediction model. On the synthetic dataset, the XGBoost model was found to accurately capture individuals' skill retention curves, where the features that contributed most (21%) to the prediction model's accuracy were found to be the considered learning curve parameters. Overall, this paper shows that experiment data of skill-based tracking tasks can be used to predict skill decay curves using XGBoost, but that more research and data are needed to achieve sufficient accuracy and reliability at an individual level for practical applications.