For transitions of control in automated vehicles, driver monitoring systems (DMS) may need to discern task difficulty and driver preparedness. Such DMS require models that relate driving scene components, driver effort, and eye measurements. Across two sessions, 15 participants enacted receiving control within 60 randomly ordered dashcam videos (3-second duration) with variations in visible scene components: road curve angle, road surface area, road users, symbols, infrastructure, and vegetation/trees while their eyes were measured for pupil diameter, fixation duration, and saccade amplitude. The subjective measure of effort and the objective measure of saccade amplitude evidenced the highest correlations (r = 0.34 and r = 0.42, respectively) with the scene component of road curve angle. In person-specific regression analyses combining all visual scene components as predictors, average predictive correlations ranged between 0.49 and 0.58 for subjective effort and between 0.36 and 0.49 for saccade amplitude, depending on cross-validation techniques of generalization and repetition. In conclusion, the present regression equations establish quantifiable relations between visible driving scene components with both subjective effort and objective eye movement measures. In future DMS, such knowledge can help inform road-facing and driver-facing cameras to jointly establish the readiness of would-be drivers ahead of receiving control.@en

Objective: We investigated a driver monitoring system (DMS) designed to adaptively back up distracted drivers with automated driving. Background: Humans are likely inadequate for supervising today’s on-road driving automation. Conversely, backup concepts can use eye-tracker DMS to retain the human as the primary driver and use computerized control only if needed. A distraction DMS where perceived false alarms are minimized and the status of the backup is unannounced might reduce problems of distrust and overreliance, respectively. Experimental research is needed to assess the viability of such designs. Methods: In a driving simulator, 91 participants either supervised driving automation (auto-hand-on-wheel vs. auto-hands-off-wheel), drove with different forms of DMS-induced backup control (eyes-only-backup vs. eyes-plus-context-backup; visible-backup vs. invisible-backup), or drove without any automation. All participants performed a visual N-back task throughout. Results: Supervised driving automation increased visual distraction and hazard non-responses compared to backup and conventional driving. Auto-hand-on-wheel improved response generation compared to auto-hands-off-wheel. Across entire driving trials, the backup improved lateral performance compared to conventional driving. Without negatively impacting safety, the eyes-plus-context-backup DMS reduced unnecessary automated control compared to the eyes-only-backup DMS conditions. Eyes-only-backup produced low satisfaction ratings, whereas eyes-plus-context-backup satisfaction was on par with automated driving. There were no appreciable negative consequences attributable to the invisible-backup driving automation. Conclusions: We have demonstrated preliminary feasibility of DMS designs that incorporate driving context information for distraction assessment and suppress their status indication. Application: An appropriately designed DMS can enable benefits for automated driving as a backup.

@en

This work aimed to organise recommendations for keeping people engaged during human supervision of driving automation, encouraging a safe and acceptable introduction of automated driving systems. First, heuristic knowledge of human factors, ergonomics, and psychological theory was used to propose solution areas to human supervisory control problems of sustained attention. Driving and non-driving research examples were drawn to substantiate the solution areas. Automotive manufacturers might (1) avoid this supervisory role altogether, (2) reduce it in objective ways or (3) alter its subjective experiences, (4) utilize conditioning learning principles such as with gamification and/or selection/training techniques, (5) support internal driver cognitive processes and mental models and/or (6) leverage externally situated information regarding relations between the driver, the driving task, and the driving environment. Second, a cross-domain literature survey of influential human-automation interaction research was conducted for how to keep engagement/attention in supervisory control. The solution areas (via numeric theme codes) were found to be reliably applied from independent rater categorisations of research recommendations. Areas (5) and (6) were addressed by around 70% or more of the studies, areas (2) and (4) in around 50% of the studies, and areas (3) and (1) in less than around 20% and 5%, respectively. The present contribution offers a guiding organisational framework towards improving human attention while supervising driving automation.

@en

This PhD thesis document is a collection of several of my published (and submitted) peer review journal articles from underneath the Human Factors of Automated Driving (HF Auto, PITN-GA-2013-605817) seventh framework program (FP7) of the European Commission. The topics include: human factors, automotive road safety, autonomous/automated driving technology, human supervisory control, adaptive automation, driver state monitoring, and scene-tied (situated) eye-based assessments of attention. Outside of the publications are summary, introduction, and conclusion chapters as well as contribution appendices to tie all the related work together. Summary: Like fatigue and distraction driving aids before, the advent of additional driving automation/autonomy poses new challenges for protecting road users now against vigilance decrements. Within the larger Human Factors of Automated Driving (HFAuto) project, the goal of this thesis was ‘to develop a system that is able to monitor the driver’s vigilance’. The approach taken was to investigate vigilance from a cognitive systems engineering (ecological perspective). Instead of conceptually restricting vigilance to be some kind of internal cognitive state/property of a driver, this thesis treated vigilance as a state/property of a system (i.e., the relationship between a driver and driving scene/situation). This thesis contains seven research papers in the form of literature reviews and experiments with eye-tracking, driving video clips, driving simulation, and on-road semi-naturalistic observation. It can be concluded form this thesis, that to develop driver monitoring systems (DMS) of driving vigilance, eye measurements (especially of movement distances) and scene contents (especially road curvatures and collision hazards) are important and relatable factors. Furthermore, it is concluded that these factors are obtainable in viable ways for future research and development application efforts. Specifically, the studies suggest means for DMS to be targeted to protect and maintain a foundational level or inner-most loop of driving attention at a behavioral level (rather than interactive implicit cognitive layers and representational experiences that can be added on top). An applied observational, data-driven, and behavioral/situated approach is expected to better avoid higher order cognitive ambiguity/dilemmas, and so serves to make more end-user acceptable DMS more tractable.@en

The topic of situation awareness has received continuing interest over the last decades. Freeze-probe methods, such as the Situation Awareness Global Assessment Technique (SAGAT), are commonly employed for measuring situation awareness. The aim of this paper was to review validity issues of the SAGAT and examine whether eye movements are a promising alternative for measuring situation awareness. First, we outlined six problems of freeze-probe methods, such as the fact that freeze-probe methods rely on what the operator has been able to remember and then explicitly recall. We propose an operationalization of situation awareness based on the eye movements of the person in relation to their task environment to circumvent shortfalls of memory mediation and task interruption. Next, we analyzed experimental data in which participants (N = 86) were tasked to observe a display of six dials for about 10 min, and press the space bar if a dial pointer crossed a threshold value. Every 90 s, the screen was blanked and participants had to report the state of the dials on a paper sheet. We assessed correlations of participants’ task performance (% of threshold crossing detected) with visual sampling scores (% of dials glanced at during threshold crossings) and freeze-probe scores. Results showed that the visual-sampling score correlated with task performance at the threshold-crossing level (r = 0.31) and at the individual level (r = 0.78). Freeze-probe scores were low and showed weak associations with task performance. We conclude that the outlined limitations of the SAGAT impede measurement of situation awareness, which can be computed more effectively from eye movement measurements in relation to the state of the task environment. The present findings have practical value, as advances in eye-tracking cameras and ubiquitous computing lessen the need for interruptive tests such as SAGAT. Eye-based situation awareness is a predictor of performance, with the advantage that it is applicable through real-time feedback technologies.

@en

Background Automated driving is often proposed as a solution to human errors. However, fully automated driving has not yet reached the point where it can be implemented in real traffic. This study focused on adaptively allocating steering control either to the driver or to an automated pilot based on momentary driver distraction measured from an eye tracker. Methods Participants (N = 31) steered a simulated vehicle with a fixed speed, and at specific moments were required to perform a visual secondary task (i.e., changing a CD). Three conditions were tested: (1) Manual driving (Manual), in which participants steered themselves. (2) An automated backup (Backup) condition, consisting of manual steering except during periods of visual distraction, where the driver was backed up by automated steering. (3) A forced manual drive (Forced) condition, consisting of automated steering except during periods of visual distraction, where the driver was forced into manual steering. In all three conditions, the speed of the vehicle was automatically kept at 70 km/h throughout the drive. Results The Backup condition showed a decrease in mean and maximum absolute lateral error compared to the Manual condition. The Backup condition also showed the lowest self-reported workload ratings and yielded a higher acceptance rating than the Forced condition. The Forced condition showed a higher maximum absolute lateral error than the Backup condition. Discussion In conclusion, the Backup condition was well accepted, and significantly improved performance when compared to the Manual and Forced conditions. Future research could use a higher level of simulator fidelity and a higher-quality eye-tracker. @en

Across the automotive industry, manufacturers have recently released various Partial Automation systems (SAE Level 2) which allow simultaneous/combined execution of both lateral and longitudinal vehicle control at the same time, yet still require active human supervision/engagement. Current reactive trends will be reviewed across major automotive players regarding differences in terminology, HMI input/outputs, and escalation intervals. Scholarly research is also reviewed pertaining to proactive strategies for driver engagement. Additionally, human factors research and findings will be presented regarding recommendations for situation awareness, human machine interfaces, TOR, as well as shared control concepts. The tutorial will conclude with discussion and brainstorming around outlook toward tele-operated remote driving services (Tele-Driving); what they have to offer beyond assisted/automated driving, autonomous vehicles, and ride-hailing/car-sharing paradigms; as well as the design/conduct of human factors research regarding Tele-Driving.

@en

A common challenge with processing naturalistic driving data is that humans may need to categorize great volumes of recorded visual information. By means of the online platform CrowdFlower, we investigated the potential of crowdsourcing to categorize driving scene features (i.e., presence of other road users, straight road segments, etc.) at greater scale than a single person or a small team of researchers would be capable of. In total, 200 workers from 46 different countries participated in 1.5. days. Validity and reliability were examined, both with and without embedding researcher generated control questions via the CrowdFlower mechanism known as Gold Test Questions (GTQs).By employing GTQs, we found significantly more valid (accurate) and reliable (consistent) identification of driving scene items from external workers. Specifically, at a small scale CrowdFlower Job of 48 three-second video segments, an accuracy (i.e., relative to the ratings of a confederate researcher) of 91% on items was found with GTQs compared to 78% without. A difference in bias was found, where without GTQs, external workers returned more false positives than with GTQs. At a larger scale CrowdFlower Job making exclusive use of GTQs, 12,862 three-second video segments were released for annotation. Infeasible (and self-defeating) to check the accuracy of each at this scale, a random subset of 1012 categorizations was validated and returned similar levels of accuracy (95%).In the small scale Job, where full video segments were repeated in triplicate, the percentage of unanimous agreement on the items was found significantly more consistent when using GTQs (90%) than without them (65%). Additionally, in the larger scale Job (where a single second of a video segment was overlapped by ratings of three sequentially neighboring segments), a mean unanimity of 94% was obtained with validated-as-correct ratings and 91% with non-validated ratings. Because the video segments overlapped in full for the small scale Job, and in part for the larger scale Job, it should be noted that such reliability reported here may not be directly comparable. Nonetheless, such results are both indicative of high levels of obtained rating reliability.Overall, our results provide compelling evidence for CrowdFlower, via use of GTQs, being able to yield more accurate and consistent crowdsourced categorizations of naturalistic driving scene contents than when used without such a control mechanism. Such annotations in such short periods of time present a potentially powerful resource in driving research and driving automation development.

@en

Human factors researchers are well familiar with Sheridan and Verplank’s (1978) ‘levels of automation’. Although this automation dimension has proved useful, the last decade has seen a vast increase of automation in different forms, especially in transportation domains. To capture these and future developments, we propose an extended automation taxonomy via additional dimensions. Specifically, we propose a 4D LINT representation for vehicle operation regarding control across multiple simultaneous dimensions of (1) Location (from local to remote), (2) Identity (between human and computer), (3) Number of agents (degree of centralization of control), as well as (4) adaptive optimization over Time. Our model aims to provide guidance and support in communicable ways to allocation authority agents (whether human or computer) in optimized supervisory outer loop control of complex and intelligent dynamic systems for more efficient, safe, and robust transportation operations@en

In pioneering work, Senders (1983) tasked five participants to watch a bank of six dials, and found that glance rates and times glanced at dials increase linearly as a function of the frequency bandwidth of the dial's pointer. Senders did not record the angle of the pointers synchronously with eye movements, and so could not assess participants’ visual sampling behavior in regard to the pointer state. Because the study of Senders has been influential but never repeated, we replicated and extended it by assessing the relationship between visual sampling and pointer state, using modern eye-tracking equipment. Eye tracking was performed with 86 participants who watched seven 90-second videos, each video showing six dials with moving pointers. Participants had to press the spacebar when any of the six pointers crossed a threshold. Our results showed a close resemblance to Senders’ original results. Additionally, we found that participants did not behave in accordance with a periodic sampling model, but rather were conditional samplers, in that the probability of looking at a dial was contingent on pointer angle and velocity. Finally, we found that participants sampled more in agreement with Nyquist sampling when the high bandwidth dials were placed in the middle of the bank rather than at its outer edges. We observed results consistent with the saliency, effort, expectancy, and value model and conclude that human sampling of multidegree of freedom systems should not only be modeled in terms of bandwidth but also in terms of saliency and effort.

@en

Research indicates that crashes between a cyclist and a car often occur even when the cyclist must have seen the approaching car, suggesting the importance of hazard anticipation skills. This study aimed to analyze cyclists’ eye movements and crossing judgments while approaching an intersection at different speeds. Thirty-six participants watched animated video clips with a car approaching an uncontrolled four-way intersection and continuously indicated whether they would cross the intersection first. We varied (1) car approach scenario (passing, colliding, stopping), (2) traffic complexity (one or two approaching cars), and (3) cyclist’s approach speed (15, 25, or 35 km/h). Results showed that participants looked at the approaching car when it was relevant to the task of crossing the intersection and posed an imminent hazard, and they directed less attention to the car after it had stopped or passed the intersection. Traffic complexity resulted in divided attention between the two cars, but participants retained most visual attention to the car that came from the right and had right of way. Effects of cycling speed on cyclists’ gaze behavior and crossing judgments were small to moderate. In conclusion, cyclists’ visual focus and crossing judgments are governed by situational factors (i.e., objects with priority and future collision potential), whereas cycling speed does not have substantial effects on eye movements and crossing judgments.@en

A majority (95%) of crashes can be attributed to humans, with the highest cause category (41%) involving errors of recognition (i.e., inattention, distraction, inadequate surveillance) [1]. Driving safety research often claims that as much as 90% of the information that drivers use is visual. However, these claims have been hampered by a lack of numerical measurement systems [2]. Presently, we develop an ordinal visual driving scene complexity measurement based on human judgments and eye behavior. Mimicking the rebuilding of situation awareness in take-over conditions we presented 60 randomly ordered video clips (3 s duration), varying complexity factors of traffic density, road curvature, and miscellaneous visual features. Eyes of 15 participants were recorded while viewing the clips, and participants rated “how much effort for you to take control and drive within that segment?” on a 100 point scale. Effort ratings showed a monotonic increase with the number of complexity factors present. A statistically significant increase was also found for saccade amplitude, whereas a statistically significant decrease was found for fixation duration. Pupil size also showed a significant increase but only between 2 complexity levels and at a relatively less convincing strength. In conclusion, the present complexity factor coding scheme apparently corresponds to subjective effort. Further consideration should be given to relating eye tracking measures to visual driving scene components and task demands. In real-time driving systems, both human occupant(s) and computerized processes may observe the same scene at the same time, and matching the machine quantification of the situation to intuitive human judgments is expected to aid in the adherence to advisories and acceptance of automated aids.@en

The “drivenger” aim of the current study was to investigate attentional differentiation of drivers (who are in control) from passengers (who have no control) to inform future driver-in-the-loop monitoring/detection systems and facilitate multiple levels of manual/automated driving. Eye-tracking glasses were worn simultaneously by the driver and front seat passenger on 32 on road trips. Halfway en-route, the passenger was tasked with pretending with their eyes to be driving. Converging with a recent and independent drivenger study, our results found differences of higher probabilities of small saccades and significantly shorter blinks from our drivers and pseudo-drivers. Additionally, a new measure of eye eccentricity differentiated between driver/passenger roles. While naturalistic attentional manipulations may not be appropriately safe/available with actual automated vehicles, future studies might aim to further use the eye behavior of passengers to refine robust measures of driver (in)attention with increasing reductions in measurement intrusiveness and data filtering/processing overhead requirements.@en

Objective
This review aimed to characterize tasks applied in driving research, in terms of instructions/conditions, signal types/rates, and component features in comparison to the classic vigilance literature.

Background
Driver state monitoring is facing increased attention with evolving vehicle automation, and real-time assessment of driver vigilance could provide widespread value across various levels (e.g., from monitoring the alertness of manual drivers to verifications of readiness in transitions of control between automated and manual driving). However, task requirement comparisons between the classic vigilance research and vigilance in car driving have not to date been systematically conducted.

Method
This study decomposed the highest-cited vigilance literature of each full decade since the 1940s for the situational features of the renowned vigilance decrement phenomenon originating from Mackworth (1948). A consensus set of 18 different situational features was compiled and included for example an (1) isolated (2) subject … perceiving (3) rare (4) signals … against (10) frequent (11) noise … in a (17) prolonged (18) task. Next, we reviewed 69 experimental vigilance task operationalizations (i.e., required signal detection and response) within 39 publications concerned with driving vigilance. All vigilance tasks were coded as “driving vigilance tasks” or “non-driving vigilance tasks” based on the perceptual signal and response action both belonging to normal driving activity or not. Presence, absence, and unreported presence/absence of each of the 18 features was rated for each task respectively as “overlap”, “contrary”, and “unspecified”. In conjunction, instructions/environmental conditions, signal definitions, signal rates, and summaries of the experimental vigilance tasks were extracted.

Results
A majority of driving vigilance tasks was performed in simulators (69%) compared to on-road (28%) and watching videos (3%) along with large differences in task conditions. Participants had to maintain fixed speed/lane positions in the simulators in higher proportion (74%) than on the road (36%) where they had only to drive “normally” and/or by loose conventions like “according to the law” more often (55% versus 15%). Additionally, presence of other traffic was found more often on-road (91%) than in simulators (48%). A specification of signals to detect and react to was found present within/for driving less often (59%) than alongside/in conjunction with driving (100%). Likewise, rates of signals (i.e., frequency of signal occurrence) were reported more often for non-driving vigilance tasks (80%) than in driving vigilance tasks (21%). For driving vigilance tasks, the highest overlap was 12 of the 18 features present (67%). On average, results showed relatively low levels of classic feature overlap (36%) with high rates of unspecified feature presence (46%) for driving vigilance tasks compared to non-driving vigilance tasks with higher classic feature overlap (64%) and fewer features unspecified (13%).

Conclusion and application
There is little overlap between the well-known and often cited vigilance decrement phenomenon and published experimental tasks of driving vigilance. Major differences were also found in the instructions/environmental conditions of simulator versus on-road experimental driving vigilance tasks. What driving vigilance practically is in the real-world thus remains a promising area for future research. We recommend that researchers apply approaches which account for more real-world driving features to better expose and address uncertainty regarding driving and vigilance.@en

The topic of transitions in automated driving is becoming important now that cars are automated to ever greater extents. This paper proposes a theoretical framework to support and align human factors research on transitions in automated driving. Driving states are defined based on the allocation of primary driving tasks (i.e., lateral control, longitudinal control, and monitoring) between the driver and the automation. A transition in automated driving is defined as the process during which the human-automation system changes from one driving state to another, with transitions of monitoring activity and transitions of control being among the possibilities. Based on ‘Is the transition required?’, ‘Who initiates the transition?’, and ‘Who is in control after the transition?’, we define six types of control transitions between the driver and automation: (1) Optional Driver-Initiated Driver-in-Control, (2) Mandatory Driver-Initiated Driver-in-Control, (3) Optional Driver-Initiated Automation-in-Control, (4) Mandatory Driver-Initiated Automation-in-Control, (5) Automation-Initiated Driver-in-Control, and (6) Automation-Initiated Automation-in-Control. Use cases per transition type are introduced. Finally, we interpret previous experimental studies on transitions using our framework and identify areas for future research. We conclude that our framework of driving states and transitions is an important complement to the levels of automation proposed by transportation agencies, because it describes what the driver and automation are doing, rather than should be doing, at a moment of time.@en

Automated driving vehicles of the future will most likely include multiple modes and levels of operation and thus include various transitions of control (ToC) between human and machine. Traditional activation devices (e.g., knobs, switches, buttons, and touchscreens) may be confused by operators among other system setting manipulators and also susceptible to inappropriate usage. Non-intrusive eye-tracking measures may assess driver states (i.e., distraction, drowsiness, and cognitive overload) automatically to trigger manual-to-automation ToC and serve as a driver readiness verification during automation-to-manual ToC. Our integrated driver state monitor is overviewed here within the scope of this brief system description/demonstration paper. It combines gaze position, gaze variability, eyelid opening, as well as external environmental complexity from the driving scene to facilitate ToC in automated driving. As both driver facing and forward facing cameras become increasingly commonplace and even legally mandated within various automated driving vehicles, our integrated system helps inform relevant future research and development towards improved human-computer interaction and driving safety.

@en

Background: Recent advances in the growing domain of automated driving suggest the need for thoughtful design of human-computer interaction strategies. For example, human drivers can process scene variability on implicit levels, but automated systems require explicit rule-based judgments of similarity and difference. What level of abstraction an automation uses in its visual perception may mean the difference between effective human-automation communication, or “uncanny valley”-like conflicts leading to problems of automation disuse, misuse, or abuse. Purpose of study: In the present research, different quantifications (semantic coding vs. computer vision features) of driving scene-to-scene similarity and difference were compared against intuitive human judgments as a reference point for future human-automation interactions.@en

The assessment of mental workload could be helpful to road safety especially if developments of vehicle automation will increasingly place drivers into roles of supervisory control. With the rapidly decreasing size and increasing resolution of cameras as well as exponential computational power gains, remote eye measurements are growing in popularity as non-obtrusive and non-distracting tools for assessing driver workload. This review summarizes literature on the relation between eye measurement parameters and drivers’ mental workload. Various eye activity measures including blinks, fixations, and saccades have previously researched and confirmed as useful estimates of a driver's mental workload. Additionally, recent studies in pupillometry have shown promise for real-time prediction and assessment of driver mental workload after effects of illumination are accounted for. Specifically, workload increases were found to be indicated by increases in blink latency, PERCLOS, fixation duration, pupil dilation, and ICA; by decreases in blink duration and gaze variability; and with mixed results regarding blink rate. Given such a range of measures available, we recommend using multiple assessment methods to increase validity and robustness in driver assessment.@en