Automation errors may result in human performance issues that are often difficult to grasp. Skraaning and Jamieson (2023) proposed a taxonomy for classifying automation errors into categories based on the visible symptoms of design problems, so as to benefit the design of training scenarios. In this paper, we propose a complementary classification that is based on the mechanisms of human-automation interaction guided by Rasmussen’s Skill, Rule and Knowledge (SRK) taxonomy. We identified four main failure classes and expect that this classification can support automation designers.@en

Previous research indicated a need to improve pilot training with regard to understanding of autopilot logic and behavior, especially in non-routine situations. Therefore, we tested the effect of problem-based exploratory training on pilots’ understanding of autopilot functions. Using a moving-base flight simulator, general aviation pilots (n = 45) were trained to diagnose failures either without foreknowledge and guidance (exploratory group), without foreknowledge but with some guidance (exploratory-guidance group) or with foreknowledge and full guidance (control group). They subsequently performed six test scenarios in which their understanding of the effects of failures was tested by requiring them to deduce the failures and select autopilot modes that were still functioning. Those who received exploratory training with guidance were significantly more likely than the other groups to diagnose failures correctly. The exploratory training group also selected the most appropriate functioning autopilot modes significantly faster than the control group. The results suggest that exploratory training with an appropriate level of guidance is useful for gaining a practical understanding of autopilot logic and behavior. Exploratory training may help to improve transfer of training to operational practice, and prevent automation surprises and accidents.

@en

Background. Previous studies and accident analyses have shown that pilots can make roll reversal errors when responding to bank angles shown by the artificial horizon in the Primary Flight Display (PFD). In the current study, we tested whether adding stereoscopic depth cues to the artificial horizon may lead to better bank angle representation due to an improved figureground separation between the symbols. Method. Stereoscopic depth cues were created by using a half-silvered mirror multi-layer PFD, which presented the horizon symbol on a lower layer and the aircraft symbol on a higher layer. A group of 23 non-pilots and 18 general aviation pilots were shown left or right bank angles on this multi-layer PFD as well as on a normal single-layer PFD, with the task to roll the wings level using a joystick. Results. In the pilot group, a similar amount of roll-reversal errors was made with both displays (median = 3.3%) with no significant difference, p = 0.635. In the non-pilot group, fewer roll-reversal errors were observed with the multi-layer display, but this difference did not reach significance either (median = 3.3% vs. 5.0%, p = 0.182). In both pilots and non-pilots, the reaction time was longer in the multi-layer display, which reached significance in the non-pilots (p = 0.016) but not in pilots (p = 0.215). Participants noticed that the depth was only visible during the start of the session. Conclusions. The results suggest that using stereoscopic depth cues are not a viable manner to enhance the figure-ground relation in the artificial horizon. @en

Background. Mnemonic procedures are currently being taught to airline pilots to manage startle and surprise. We previously tested the effectiveness of a four-item mnemonic. Pilots generally rated it as useful but some remarked that it induced too much additional workload. Therefore, we tested whether a simpler mnemonic, Aviate-Breathe-Check, would be more useful. Method. The experiment took place in a hexapod simulator with a Piper Seneca aerodynamic model and a generic cockpit. Airline pilots (n = 25) were divided into an experimental (“ABC”) and control group. All received ground training on startle and surprise, which included instructions on the ABC mnemonic for the ABC group. The mnemonic aims to support prioritization of flight-path management (Aviate), followed by physiological and mental stress management (Breathe), followed by troubleshooting (Check). All pilots performed four familiarization scenarios, during which the ABC group practiced the ABC mnemonic. Two test scenarios were then performed to evaluate performance, mental effort, stress, and pilot evaluations of the ABC mnemonic. Results. The pilots’ evaluations of the ABC mnemonic were significantly higher than those were for the previously-tested mnemonic in the same scenarios. There were no significant differences between the ABC and control group in mental effort and stress, whereas there were trends towards higher mental effort and stress with the previous mnemonic. No significant effects on performance were found. Conclusions. The results suggest that the ABC mnemonic was more useful and easier to apply than a previously tested mnemonic. This is promising for the development of effective pilot training interventions for startle and surprise. @en

Background: Previous research has shown that experiencing motion stimuli negatively impacts cognitive performance. Objective: In the current study, we investigate whether this impact relates to Type-II spatial disorientation (SD), to motion stimulus magnitude, or to an interaction of these factors. Method: Stimuli for participants (n = 23) consisted of Earth-vertical yaw rotations on a rotating chair in a completely darkened room. In the surprise condition, the stimulus started with subthreshold acceleration, followed by suprathreshold deceleration to a non-zero velocity, inducing a sensation of rotation that is opposite to the actual rotation revealed when the lights were switched on. In the no-surprise condition, the same changes in velocity were used, but starting from (almost) zero velocity, which induced a sensation of rotation in the same direction as the actual rotation. Participants performed a self-paced arithmetic task and measurement of their cognitive performance started after the environment was revealed. Stimulus magnitude was operationalized through higher or lower peak suprathreshold deceleration. Results: The results revealed that counting speed decreased significantly when participants were surprised, constituting a large effect size. The proportion of counting errors likewise increased significantly when participants were surprised, but only in the high-magnitude condition. Application: The findings suggest that surprise caused by the recognition of SD has an involuntary disruptive effect on cognition, which may impact performance of piloting tasks. These results are relevant when modeling motion stimuli effects on performance, and when developing SD awareness training for pilots.

@en

The current study was performed to obtain insight into the retention of combat lifesaving (CLS) skills after initial training and to compare a more individualized-style training with a more classroom-style training. We measured performance at 0 month, 2 months, and 6 months after initial training in 40 CLSers (17 individualized, 23 classroom). Each test consisted of two 20-minute scenarios with a medical mannequin to simulate combat injuries. An instructor scored the actions, which were divided into critical and non-critical by medical experts. We also measured the speed of performing the protocol and perceived mental effort and anxiety. There were no differences between the groups in critical actions. The full sample made on average almost six critical errors per scenario at 6 months. However, on non-critical actions, the individualized group scored better at 0 month. The individualized group also performed the protocol faster at each test. The classroom group reported an increase in mental effort and anxiety at subsequent tests, while the individualized group did not. Based on the high number of critical errors at 6 months, and on the drop-off in performance at 2 months, we advise that extra refresher training is organized within 2 months after initial training to improve retention further down the line.

@en

Real-time physiological stress monitoring would be a relevant addition to virtual reality (VR) training for high-risk professions, such as the military. VR is highly suitable for the implementation of such monitoring due to the controlled environment and the already used wearables. However, physiological stress measurements suffer from distortion due to physical activity. Therefore, we tested whether we can use accelerometry to correct non-invasively measured heart rate (HR) for physical activity in 23 soldiers who performed three room-clearing VR scenarios. These scenarios were dynamic, in that soldiers moved around in the VR environment by walking around in the real environment. In contrast to uncorrected HR, and HR corrected by subtracting baseline HR measured when walking, the accelerometry-corrected HR was able to significantly predict the participants’ self-reported stress in the scenarios, p = 0.047, R ² = 0.11. Whereas uncorrected HR significantly predicted self-reported physical demand, p = 0.028, R ² = 0.09, the accelerometry-corrected HR did not. All HR measures significantly predicted self-reported mental effort, which was most strongly the case for uncorrected HR, p < 0.001 R ² = 0.42. These findings, in combination with the methods’ low sensitivity to motion artifacts and non-invasiveness, are very promising for its use to monitor stress in real-time during dynamic VR training scenarios.

@en

Virtual reality (VR) simulation tools, in combination with miniaturized sensor and geo-technology, represent an opportunity to prepare high-risk professionals better for uncertain situations. In the current study, we tested whether VR preparation for a police surveillance task leads to increased performance and decreased stress, workload, and a faster recuperation. Police officers (n = 46) were either prepared with a 3D interactive VR simulation of the venue, or received the standard preparation using a 2D paper-based map. Then, officers individually conducted a surveillance scenario during a real live-music concert. Position tracking, heart rate, heart rate variability (HRV), salivary cortisol levels (SCL) and self-perceived stress were assessed. Results showed that police officers with the VR preparation made less direction changes when finding target locations, had lower HRV during the surveillance, and had lower SCL during their recuperation. These results indicate that VR preparation may increase police officers’ performance and improve their recovery.@en

In the analysis of human performance and human error, considerable attention is given to the cognitive processes of actors involved in error or success scenarios. Even with awareness of hindsight bias, it takes effort to understand the actions of agents in later inspection of error scenarios. One such topic of heated discussion was the perceived poor performance of pilots in the two 737 MAX MCAS-related crashes in applying the “memory item” checklist pertaining to a runaway trim. In this paper, we argue that it is not so much the reproduction of the checklist that was lacking in these scenarios, but the trigger for even starting the checklist. Not only trim run-away problems, but several other issues likewise require an instant reaction from pilots, designated as “memory items”. Rasmussen’s simplified schematic for the “skill, rule and knowledge” taxonomy already provides the tools for properly analyzing this. The skill to provide the triggers for these reactions relies on pattern extraction from the available sensory input, and, importantly, it can only be learned in a valid training context. It is argued that re-appraisal of these items is needed, addressing explicitly the validity of the training environments that enable pilots to learn the required pattern recognition skills.@en

Ground-based demonstration of spatial disorientation (SD) has been recommended for military as well as commercial pilot training. Although the leans illusion is the most common form of SD, no data exist yet of an effective ground-based leans procedure for a hexapod simulator. In this paper we describe the development of such a procedure and its tuning with nine subjects. The procedure was then used in an experiment with 18 airline pilots, which is described elsewhere. The final procedure started with a prepositioning phase, during which the simulator platform was slowly tilted to a 3.5° prepositioning roll angle, while the pilot performed a distraction task and the instruments and outside visuals indicated level flight. Next followed an adaptation phase, during which the pilot's vestibular system adapted to the new angle, the outside visibility degraded to zero and the instruments were covered. Then the platform was moved back to level, above the perceptual threshold, after which the instruments were shown again. The pilot was then tasked to roll back to level. The addition of the motion cues caused an increase in roll reversal errors by a factor of 3 in airline pilots. The procedure can be implemented in a scenario for demonstrating the leans in a cost-effective simulator. @en

The current study explores whether different stressors in a virtual reality (VR) military training scenario cause increases in physiological stress. This would validate the use of VR simulation for stress training, as well as the physiological monitoring of trainees for educational purposes. Military cadets (n = 63) performed a patrol scenario (military convoy) in which they answered questions about their surroundings. Stressors (task difficulty, noise, lighting changes, social evaluations, electric muscle stimulation, and a simulated attack on the convoy) were stepwise added in four phases. Electrocardiogram, blood pressure, electrodermal activity, cortisol, and the cadets’ subjective threat/challenge appraisal were measured. We found that only the first phase caused a significant increase in physiological stress, as measured with heart rate, heart rate variability, and electrodermal activity. Physiological stress appeared to stay high in the second phase as well, but decreased to baseline level in the third and fourth phases, even though these phases were designed to be the most stressful. Cadets classified as threat responders based on physiological data (n = 3) scored significantly higher on subjective threat/challenge appraisal than those classified as challenge responders (n = 21). It seems that in the tested VR training scenario, the novelty of the scenario was the only effective stress stimuli, whereas the other implemented stressors did not cause a measurable physiological response. We conclude that if VR training scenarios are to be used for stress training, these should confront trainees with unpredictable but context-specific demands.@en

Previous studies show that pilots sometimes make roll reversal errors (RREs) when responding to the aircraft bank angle shown on the attitude indicator (AI). This is suggestive of a perceptual ambiguity in the AI. In the current study, we investigated whether expectation contributes to such misperception. Twenty nonpilots performed tasks in a fixed-base flight simulator. Their expectation about the bank angle was manipulated with a flying task using outside view only. When flying at a bank angle, the outside view disappeared, a moving-horizon type AI was shown, and participants had to roll the wings level, trusting the AI. The AI often matched the previously flown turn. However, in some runs, it showed an opposite bank direction (Opposite condition), which was hypothesized to facilitate a misperception. In some other runs, it showed level flight (Level condition), which should not facilitate this. In a second session, participants rolled wings level without preceding flying task (Baseline condition). Participants made 11.2 times more RREs in the Opposite condition (75% error rate) compared to Baseline condition (6.7%), and 2.5 times more compared to the Level condition (30%). This indicates that RREs were in many cases caused by expectation-induced misperception of the AI bank angle.

@en

Objective: We tested whether a procedure in a hexapod simulator can cause incorrect assumptions of the bank angle (i.e., the “leans”) in airline pilots as well as incorrect interpretations of the attitude indicator (AI). Background: The effect of the leans on interpretation errors has previously been demonstrated in nonpilots. In-flight, incorrect assumptions can arise due to misleading roll cues (spatial disorientation). Method: Pilots (n = 18) performed 36 runs, in which they were asked to roll to wings level using only the AI. They received roll cues before the AI was shown, which matched with the AI bank angle direction in most runs, but which were toward the opposite direction in a leans-opposite condition (four runs). In a baseline condition (four runs), they received no roll cues. To test whether pilots responded to the AI, the AI sometimes showed wings level following roll cues in a leans-level condition (four runs). Results: Overall, pilots made significantly more errors in the leans-opposite (19.4%) compared to the baseline (6.9%) or leans-level condition (0.0%). There was a pronounced learning effect in the leans-opposite condition, as 38.9% of pilots made an error in the first exposure to this condition. Experience (i.e., flight hours) had no significant effects. Conclusion: The leans procedure was effective in inducing AI misinterpretations and control input errors in pilots. Application: The procedure can be used in spatial disorientation demonstrations. The results underline the importance of unambiguous displays that should be able to quickly correct incorrect assumptions due to spatial disorientation.

@en

Background: Mnemonic-type startle and surprise procedures were previously proposed to help pilots cope with startle and surprise in-flight, but effects on performance after procedure execution have not yet been investigated. Objective: Thus, we tested the effectiveness a new mnemonic-type procedure in a moving-base simulator with a non-linear model of a small twin-propeller aircraft flown single-pilot. Method: An experimental group of twelve line pilots was trained to use a four-item procedure: 1. Calm down: take a deep breath, sit up straight and relax shoulders and hands. 2. Observe: call out the basic flight parameters. 3. Outline: formulate a hypothesis about the problem. 4. Lead: formulate and execute a plan of action. A control group of twelve line pilots received a control training. Next, all pilots performed four scenarios with startling and surprising events. Data were obtained on pilot performance, stress, procedure application and evaluation. Results: Application of the procedure in the test scenarios was high (90.0% full, 100.0% partly), and pilots evaluated the procedure positively (median: 4 on a 1–5 point scale). There was significantly superior decision-making in the experimental group, but immediate responses were significantly less optimal. Pilots sometimes applied the procedure at inappropriate moments. Conclusion: The results of the tested mnemonic-type procedure were promising. The procedure may benefit, however, from modifications to reduce complexity and to stimulate application at the appropriate moment.

@en

We hypothesized that an incorrect expectation due to spatial disorientation may induce roll reversal errors. To test this, an in-flight experiment was performed, in which forty non-pilots rolled wings level after receiving motion cues. A No-leans condition (subthreshold motion to a bank angle) was included, as well as a Leans-opposite condition (leans cues, opposite to the bank angle) and a Leans-level condition (leans cues, but level flight). The presence of leans cues led to an increase of the roll reversal error (RRE) rate by a factor of 2.6. There was no significant difference between the Leans-opposite and Leans-level condition. This suggests that the expectation strongly affects the occurrence of an RRE, and that people tend to base their responses on motion cues instead of on information on the AI. We conclude that expectation and spatial disorientation have a large effect on piloting errors and may cause hazardous aircraft upsets.
@en

After several recent flight safety events, such as the accident of Air France flight 447 in 2009, investigators determined that surprise and startle can severely disrupt pilot responses. They concluded that pilots need to be better prepared for unexpected and potentially startling situations. In response, aviation safety authorities have recommended and mandated that startle and surprise should receive more attention in pilot training. However, there is insufficient scientific data available on pilots’ behavior in startling and surprising situations, and on how they can best be trained for these situations. This thesis addresses this problem, by studying startle and surprise in pilots, and by investigating which training interventions can strengthen the pilots’ response to unexpected situations...@en

Several checklist-based methods have been proposed to help pilots manage startle in unexpected situations. In the current experiment, we tested how pilots reacted to using such a method, which featured the mnemonic COOL: Calm down – Observe – Outline – Lead. Using a motion-based simulator outfitted with a non-linear aerodynamic model of a small twin-propeller aircraft, twelve pilots practiced using the COOL method before performing four test scenarios involving startling events. Application of the full method in the test scenarios was high (90-100%), and pilots rated the method on average as useful (4 on a 1-5 point Likert scale). The first two steps of the method were seen as the “core” of the method. However, pilots also displayed difficulty with prioritizing dealing with immediate threats over executing the method. The results are promising, but they also warn us to be cautious when introducing a startle management method.@en

Aviation safety organizations have recommended that airline pilots are trained for startle and surprise. However, little information is available on useful training interventions. Therefore, a training intervention trial was executed during airline recurrent simulator training. The method consisted of a slow visual scan from the side-window, over the instruments, ending with facing the other pilot. Following a recorded video instruction, 38 airline pilots in two-pilot crews performed a training scenario in which they could apply the method. Data on application and evaluation of the method were obtained from each pilot. Few pilots actually applied the method (18.4%), and many gave low ratings to applicability of the method in the scenario, as well as in operational practice. Results show that a startle management method, as well as manner in which it is trained, should be carefully evaluated before being implemented in training practice.@en

Objective: This study tested whether simulator-based training of pilot responses to unexpected or novel events can be improved by including unpredictability and variability in training scenarios. Background: Current regulations allow for highly predictable and invariable training, which may not be sufficient to prepare pilots for unexpected or novel situations in-flight. Training for surprise will become mandatory in the near future. Method: Using an aircraft model largely unfamiliar to the participants, one group of 10 pilots (the unpredictable and variable [U/V] group) practiced responses to controllability issues in a relatively U/V manner. A control group of another 10 pilots practiced the same failures in a highly predictable and invariable manner. After the practice, performance of all pilots was tested in a surprise scenario, in which the pilots had to apply the learned knowledge. To control for surprise habituation and familiarization with the controls, two control tests were included. Results: Whereas the U/V group required more time than the control group to identify failures during the practice, the results indicated superior understanding and performance in the U/V group as compared to the control group in the surprise test. There were no significant differences between the groups in surprise or performance in the control tests. Conclusion: Given the results, we conclude that organizing pilot training in a more U/V way improves transfer of training to unexpected situations in-flight. Application: The outcomes suggest that the inclusion of U/V simulator training scenarios is important when training pilots for unexpected situations.

@en

We discuss an extension of the basic principles underlying the human haptic just noticeable difference (JND) in perceiving a manipulator's mechanical properties from force feedback. Two cases are studied: 1) the JND in perceiving the stiffness of manipulators with various masses; 2) the JND in perceiving the damping of a combined mass-spring-damper system with varying stiffness and mass. The extended JND laws are obtained through mapping psychophysical findings to JND formulations based on frequency response functions. We first present two human-factor experiments in which subjects discriminated between different levels of manipulator stiffness/damping while moving the manipulator with a prescribed sinusoidal deflection. For the two testing cases both JNDs violate Weber's law: Due to the increases in mass, the normalized stiffness JND (the Weber fraction) decreases as the reference stiffness level increases; The damping JND for a constant reference damping increases with higher combined responses of stiffness and mass. On the basis of weighting the frequency response magnitude of mechanical properties, we performed model identification that fit the experimental observations, and extended the JND laws for the two testing cases. Our extended JND laws indicate that: 1) stiffness and mass affect the stiffness JND in the same way, the stiffness JND is a fixed proportion of the combined frequency response of stiffness and mass; 2) the frequency response magnitude of the damping JND is a fixed proportion of the frequency response magnitude of the combined system (the mass-spring-damper system).@en