The Role of Simulated Emotions in Reinforcement Learning

Abstract

Transparency of behavior is important for robots that work with humans. If such robots need to adapt to a variety of users and tasks, they need to learn to optimize their behavior, and Reinforcement Learning (RL) is a promising learning method for this purpose. However, the behavior generated by RL is not inherently transparent due to the exploration/exploitation tradeoff that is needed to optimize a policy.Emotions are -for humans- a natural way of communicating intent and situational appraisal. In this study, we implemented emotional expressions based on Temporal Differences as a means to increase the transparency of a robot's learning process. We analysed the effect on the human teacher's behavior and experience, and on the robot's learning result and learning process.A between-subject experiment with 61 participants and three robot conditions was performed: no emotions, simulated emotions, and simulated emotions with matching attribution. The learning task was one where a human teacher had to help a humanoid robot to learn the meaning of three colors.Our results demonstrate minimal differences between these three conditions. This means that for simple tasks, emotional expressions grounded in RL do not help nor hurt. We discuss our findings and propose three important criteria for interactive learning tasks when investigating the effect of emotional expressions grounded in RL. Such tasks need to be sufficiently complex, afford robot autonomy, and the emotion must be informative about how the user could influence the robot's actions.