Capturing drivers’ affective responses given driving context and driver-pedestrian interactions remains a challenge for designing in-vehicle, empathic interfaces. To address this, we conducted two lab-based studies using camera and physiological sensors. Our first study collected participants’ (N = 21) emotion self-reports and physiological signals (including facial temperatures) toward non-verbal, pedestrian crossing videos from the Joint Attention for Autonomous Driving dataset. Our second study increased realism by employing a hybrid driving simulator setup to capture participants’ affective responses (N = 24) toward enacted, non-verbal pedestrian crossing actions. Key findings showed: (a) non-positive actions in videos elicited higher arousal ratings, whereas different in-video pedestrian crossing actions significantly influenced participants’ physiological signals. (b) Non-verbal pedestrian interactions in the hybrid simulator setup significantly influenced participants’ facial expressions, but not their physiological signals. We contribute to the development of in-vehicle empathic interfaces that draw on behavioral and physiological sensing to in-situ infer driver affective responses during non-verbal pedestrian interactions.

While affective non-verbal communication between pedestrians and drivers has been shown to improve on-road safety and driving experiences, it remains a challenge to design driver assistance systems that can automatically capture these affective cues. In this early work, we identify users' emotional self-report responses towards commonly occurring pedestrian actions while crossing a road. We conducted a crowd-sourced web-based survey (N=91), where respondents with prior driving experience viewed videos of 25 pedestrian interaction scenarios selected from the JAAD (Joint Attention for Autonomous Driving) dataset, and thereafter provided valence and arousal self-reports. We found participants' emotion self-reports (especially valence) are strongly influenced by actions including hand waving, nodding, impolite hand gestures, and inattentive pedestrian(s) crossing while engaged with a phone. Our findings provide a first step towards designing in-vehicle empathic interfaces that can assist in driver emotion regulation during on-road interactions, where the identified pedestrian actions serve as future driver emotion induction stimuli.

Automatically inferring drivers' emotions during driver-pedestrian interactions to improve road safety remains a challenge for designing in-vehicle, empathic interfaces. To that end, we carried out a lab-based study using a combination of camera and physiological sensors. We collected participants' (N=21) real-time, affective (emotion self-reports, heart rate, pupil diameter, skin conductance, and facial temperatures) responses towards non-verbal, pedestrian crossing videos from the Joint Attention for Autonomous Driving (JAAD) dataset. Our findings reveal that positive, non-verbal, pedestrian crossing actions in the videos elicit higher valence ratings from participants, while non-positive actions elicit higher arousal. Different pedestrian crossing actions in the videos also have a significant influence on participants' physiological signals (heart rate, pupil diameter, skin conductance) and facial temperatures. Our findings provide a first step toward enabling in-car empathic interfaces that draw on behavioural and physiological sensing to in situ infer driver emotions during non-verbal pedestrian interactions.

With the increasing availability of head-mounted displays (HMDs) that show immersive 360° VR content, it is important to understand to what extent these immersive experiences can evoke emotions. Typically to collect emotion ground truth labels, users rate videos through post-experience self-reports that are discrete in nature. However, post-stimuli self-reports are temporally imprecise, especially after watching 360° videos. In this work, we design six continuous emotion annotation techniques for the Oculus Rift HMD aimed at minimizing workload and distraction. Based on a co-design session with six experts, we contribute HaloLight and DotSize, two continuous annotation methods deemed unobtrusive and easy to understand. We discuss the next challenges for evaluating the usability of these techniques, and reliability of continuous annotations.