Given the expected increase of automation in the vehicles of the future, touchscreens are expected to be used in a wide variety of scenarios, including the ones that can become safety-critical.
One disadvantage of the direct interaction approach that characterizes touchscreen operation is the feedthrough of accelerations through the human body that can induce unwanted activations, leading to unsafe situations. The current paper investigates the feasibility of applying a method that can track the biodynamic responses of the elbow, wrist and index finger using cameras in a stereo configuration and an open-source pose estimator (OpenPose). An experiment with six participants was performed to understand whether different neuromuscular settings, task instructions or degrees of arm extension induce an adaptive behavior of the biodynamic feedthrough to the recorded joints. The outcomes of the experiments showed that the finger is the body part that exhibits the most adaptive behavior in terms of feedthrough, being dependent on whether it is interacting with the screen with no velocity, moving on the screen or not touching the screen. Moreover, the difference between keeping the upper limb relaxed or stiff would need to be taken into account for the specified body part when the finger is not touching the screen, showing a decrease of 29% in the mean RMS of the vertical feedthrough component when the stiff condition is exhibited. The adaptive nature of the biodynamic feedthrough was also demonstrated concerning two different cases of arm extension (close and far from the body). This study proved the feasibility of using a linear mass-spring-damper model to the data gathered with the camera-based system to describe the feedthrough effects for all the body parts, neuromuscular settings and tasks, being able to explain on average at least 75% of the variance of the raw signals. The approach presented in this study can be further refined toward reaching real-time capabilities, which is a key step in accurately identifying the highly adaptive and subjective nature of the feedthrough of the accelerations.

Aerodynamic stall has been a critical factor in recent aircraft crashes, leading to revisions in the regulations on the fidelity of stall models in flight simulation training devices. However, the updated regulations still lack a clearly defined accuracy required for effective pilot training. To determine the required accuracy, this research investigates how the Just Noticeable Difference (JND) thresholds for the stall abruptness parameter translate from a passive, observer task to an active flying task. An experiment was performed in the SIMONA Research Simulator with 16 active pilots, who performed two separate experiments. In one experiment, a stall autopilot flew the maneuver during which a staircase procedure was used to determine the passive JND threshold. The JND thresholds of a1 = 0.11 ± 0.094 found in this experiment were lower than the JND thresholds of a1 = 0.16 ± 0.14 found in a similar experiment by previous research. In the other experiment, the method of constant stimuli was used to determine the JND threshold for the active flying scenario. A psychometric curve, based on the Gaussian cumulative distribution function, was fitted using the combined responses of the participants. The resulting psychometric function of the active experiment lies entirely to the right of the passive psychometric function, and, when comparing the 75% thresholds for both experiments, the active threshold was found to be five times higher than the passive threshold. This indicates a decreased sensitivity to changes in stall abruptness when pilots are flying a stall themselves.

Touchscreens are increasingly being integrated into modern aircraft cockpits due to their space-efficiency and adaptability. However, operating touchscreens in turbulent conditions reduces input accuracy and alters the applied force. These issues are caused by involuntary limb movements induced by external accelerations such as turbulence, called biodynamic feedthrough. Understanding and mitigating this phenomenon is crucial to be able to provide consistent haptic feedback on touchscreens. This study investigates the dynamics of biodynamic feedthrough into the applied normal force on the screen, by conducting a human-in-the-loop experiment with fifteen participants. In the experiment, participants performed both stationary and dynamic position tracking tasks on a touchscreen, while being exposed to turbulence in all three linear motion directions. It was observed that on average 63% higher normal forces were applied during stationary tasks than during dynamic tasks, likely resulting in increased arm co-contraction. Additionally, a second-order mass-spring-damper model was fitted to the obtained force data. It was found that biodynamic feedthrough into the applied normal force has the highest feedthrough levels when the direction of turbulence aligns with the direction of the applied force, and that up to 90% of the disturbance-induced normal force can be described by the proposed individualized models. The results also show the variability of dynamics between participants and therefore the importance of individualized models. These findings provide insights for future developments in adaptive haptic feedback for touchscreens, aiming to enhance the reliability and safety of touchscreen interactions in turbulent environments.

The aviation industry's reliance on automation raises concerns about pilot complacency, necessitating continuous pilot proficiency measures. To that end, real-time pilot skill feedback is vital—through alerts on declining skill levels or scalable levels of autonomy. Current cybernetic methods are limited as they assume linearity and time-invariance of human behavior and lack real-time capability. Neural Networks (NNs) offer a solution but face challenges such as high computational costs and limited generalization capability. To overcome these issues, this paper introduces a new and compact Residual Network for eXplainable Convolutional MTS Classification (RN-XCM) designed explicitly to classify pilot skill levels. Results demonstrate RN-XCM's ability to accurately classify skill levels based on 1.2 seconds of dual-axis control data, achieving a test accuracy of up to 93.50%, while requiring 50% less training time than competing NN models. It also achieves a test accuracy of 80.16% for previously unseen subjects, signifying its competence as a one-size-fits-all classifier. Notably, RN-XCM performs 17.88% better when classifying dual-axis tracking tasks over single-axis tracking tasks. Overall, the possibility of real-time feedback provided by the RN-XCM can enable quantitative evaluation of pilot control behavior, therefore enhancing safety and facilitating smoother interactions between pilots and aircraft.

Understanding individual differences in manual control cybernetics plays a principal role in personalizing human-machine systems. While much of the work in cybernetics utilizes models for the average controller, individualized models have also been explored, albeit constrained by data availability and the effectiveness of identification techniques. It has previously been theorized that human controller behavior may be defined by a cost function that the controller minimizes for a given control task. This cost function comprises weighted performance and effort terms, and the weightings have been connected to cybernetic models before, but not on an individualized basis. A proposed connection is through the equalization component of the cybernetic model, which has also been outlined as an indicator of individual differences. This claim was reinforced by the analysis of previously collected experiment data in this paper: Across three independent experiments, the equalization parameters, on average, showed 33.84% higher variation than the others and a greater impact on the cost function. Thus, this paper presents a human-controller-cost-function-centered approach to predict and generate individualized sets of cybernetic parameters. Individual cost function weightings were determined using the identified equalization parameters from earlier human-in-the-loop experiments. This process exhibited clear groupings in participants, as clusters of participants with similar equalization were not identified adequately. Individual cost function weightings were then employed to predict full sets of cybernetic parameters. The results indicated that the utilized cost function is incapable of reflecting the physical limitations of the human controller, as these parameters (time delay, neuromuscular natural frequency, and neuromuscular damping coefficient) showed 56.06%, -94.43%, and -170.34% deviation compared to the experimental values. Hence, a hybrid cybernetic data augmentation method was devised, yielding signal values within 15% of experimental data.

Adequate modeling of the unsteady aerodynamics during flow separation is critical for effective pilot training in Flight Simulation Training Devices. Over the years, a stall modeling method rooted in Kirchhoff's theory of flow separation has gained popularity due to its relative simplicity and suitability for parameter identification from flight data. This method describes the lift using a single internal flow separation point variable. A major drawback of Kirchhoff's method comes from the one-dimensionality of the flow separation point, limiting the representation of asymmetric flow separation. The goal of this work is to improve the existing Cessna Citation II dynamic stall model fidelity by applying Kirchhoff's method for each wing surface, separately. The main contribution is the identification of asymmetric flow separation development, using the flight-derived roll moment and a roll moment model based on the differential flow separation between the wing surfaces. Transformations of the flow separation variables were chosen by a Multivariate Orthogonal Functions selection algorithm to capture the stall-related nonlinearities of the roll moment, yaw moment, and lateral force. The longitudinal model structures are adopted from the existing, validated baseline stall model. The lateral-directional model outputs are in good agreement with the validation flight data, showing an average reduction of 48\% in Mean Squared Error (MSE) compared to the baseline stall model. In contrast, the longitudinal model output results in an average MSE increase of 88\%, suggesting that the estimated asymmetric flow separation parameters are unsuitable for longitudinal stall modeling. A promising way to incorporate the benefits of the proposed method is suggested, by adopting a hybrid approach that combines separate sets of flow separation parameters --- symmetric and asymmetric variants --- for the longitudinal and lateral-directional models, respectively.

Although the roles of visual and haptic cues in motor tasks have been well studied, the benefits of audio cues in complex motor tasks have been underexplored. Audiovisual cues reduce reaction times and decrease the variance of endpoint responses in simple head-orienting localization tasks. The aim of this research is to explore the potential of audiovisual stimuli for motor tasks performed in a real-life environment. Participants performed simple head and arm localization tasks in an echoic room. Comparing audiovisual stimuli to unimodal stimuli, a decrease in reaction times was observed. For head responses, reaction times were reduced on average by 58 ms for visual stimuli and 35 ms for auditory stimuli. For arm responses, reaction times showed a decrease of 30 ms on average for both unimodal conditions. Likewise, we noted decreased endpoint variance in audiovisual responses in comparison to unimodal auditory responses. Moreover, we observed a modality-effector dependency in the weighting of auditory and visual signals in the audiovisual responses. These results show the potential of audiovisual stimuli for real-life applications in motor tasks.

Recent aircraft have seen the implementation of touchscreens (TSCs) on the flight deck, in sight of more intuitive and direct human-machine interactions. Biodynamic feedthrough (BDFT), i.e., the transfer of the aircraft’s accelerations through the pilot’s body to the control inputs, is however still cause for concern, preventing safe and reliable use of TSCs in turbulence. This paper describes a simulator experiment evaluating the performance in turbulent flight of model-based mitigation of BDFT occurring with TSC dragging task. Placing a TSC in front of the pilot, various motion perturbations were tested on the heave axis: multisine signals resembling turbulence, stationary (Gaussian) and variable (patchy) simulated turbulence, at three intensity levels. The results show that on average over 87% accuracy can be achieved in the identification of a personalized BDFT model at intensity of 0.75 and 0.5 m/s2 (RMS heave accelerations), decreasing to 74% for RMS intensity of 0.25 m/s2, symptom of a lower amount of feedthrough in the TSC input at low turbulence intensity. No model canceling over 70% of the BDFT components of the TSC inputs could be generalized across intensities, as the damping of the BDFT dynamics shows a 49% decrease between high and low intensities. In regards to BDFT mitigation in turbulence, models were identified from BDFT in Gaussian turbulence with accuracy comparable to models identified in multisine motion disturbances, with only 3.5% lower performance on average. Comparison between the Gaussian and patchy turbulence cases revealed a 4.7% higher BDFT mitigation performance for the former, connected to the time-varying nature of patchy turbulence. Finally, models generalizing BDFT dynamics across participants or experimental runs were found to always be outperformed by individual run models, giving up to 10% higher identification performance. These findings show that a model-based approach is promising with regards to BDFT mitigation in turbulence for TSC dragging tasks, but also that real-time identification and time-varying BDFT models might be needed to achieve consistently high mitigation performance in realistic variable turbulence.

Glaucoma impacts vision by affecting visual processing at the retinal ganglion cell level. To recreate its impact on visual processing, photobleaching has been proposed to reversibly and temporarily induce glaucoma-like saccadic reaction time. It has been established that photobleaching elevates the threshold detection levels of visual stimuli in a healthy retina. This study investigated the potential implementation of photobleaching to recreate glaucoma-like SRTs. Results from the study show that photobleaching increased the SRT and reduced the ability to detect targets within the photobleaching zone of the visual field. The elevated SRTs obtained from the study indicate that photobleaching is a valid method for obtaining mild glaucoma-like SRTs. Furthermore, the effects of photobleaching were found to be highly localized, with the impact primarily observed within the bleaching zone. Attempts to replicate the localized effects of glaucoma using discrete photobleaching were inconclusive, with elevated SRTs observed in some cases, but the effect is directly related to the total area bleached. These findings provide the relationship between photobleaching, saccadic response, and their potential implications for artificially recreating glaucoma-like elevated SRTs

To improve the safety of commercial air transport, pilots are required to train on simulators to recognize the characteristics of an impeding stall and subsequently correctly recover from it. To prevent negative training, it is important that the accuracy of the used simulation models is high. A manner to model the nonlinear, unsteady aerodynamic effects during the stall is by using Kirchhoff's theory of flow separation. However, widespread difficulties exist in correctly estimating the stall-related parameters in these models. It is not always possible to simply gather more flight data to solve this problem. Therefore, the research in this paper aims to increase model accuracy by making optimal use of already existing flight data via introduction of the slice-based modeling method. This is done by analyzing the change in the parameter estimate values when applying the system identification procedure to sliced partitions of simulated flight data. These partitions incrementally increase in size with time from the stall. The simulation data is generated to be representative of the available flight test data. The change in parameter estimates was analyzed for both the pre-stall and post-stall phase. The estimated value for each partition was compared to the actual parameter value setting in the simulation model used to create the data. Manually, an optimal window was found for each parameter for which the estimated value and actual value were equal. For the stall-related parameters this window is often not more than 10 s wider than the stall. For the linear stability and control derivatives it is found that using more data generally results in a better estimate. These window sizes were used in the estimation for each separate parameter on the real flight test data. Even though this method represents a prototype, in more than half of the validation cases a decrease in MSE of 10 % to 35 % could be achieved. This shows that the new slice-based modeling method is able to improve the accuracy of nonlinear stall models without the need to gather more flight data. Additionally, the parameter estimate behaviour analysis technique and slice-based modeling method may have applications that reach beyond the realm of stall modeling.

Technological devices are ubiquitous, think of for example smartphones and in-vehicle information systems. Both can contribute towards distracted driving where the visual field of the human controller is shifted away from the primary control task. In this paper a neural network model is trained using the InceptionTime architecture and used to detect distractions in pursuit and preview tracking tasks. For this purpose an experiment has been designed to collect data in which participants are distracted using a visual distraction called the Surrogate Reference Task (SuRT). It was found that distractions are easier to detect in tracking tasks with pursuit displays instead of preview displays. This is because in preview displays the future target trajectory is shown to the human controller, resulting in a lower tracking error compared to pursuit displays. Apart from the tracking error, the InceptionTime neural network was also trained using the time-series data of the control input and system output. Important characteristic of distracted data found were a reduced control input and higher tracking errors, which may have helped in detecting distractions. The classification models were able to predict data samples correctly with an accuracy of 80.78% and 61.66% in pursuit and preview tracking tasks with distractions, respectively. Lastly, individualised models showed better performance when compared to 'one-size-fits-all models'. Results show clear opportunities for applying neural network models in real-time to detect distractions for increasing safety in human operated machines.

While human control behavior is well-understood in continuous control tasks, little is known about how human operators detect sudden changes in the controlled element dynamics. This paper focuses on modeling this detection phase for pursuit tracking tasks. Potential triggers for the human operator to detect changes in the controlled element dynamics were investigated via a time-varying computer simulation. Based on the results, hypotheses were generated and later tested in a single-axis pursuit tracking experiment with fifteen participants, where good-quality data were collected. Transitions from approximate single to approximate double integrator dynamics and vice versa were investigated, and participants indicated if they detected the transition by pressing a button. Using the button push data, a model for each transition was developed and validated. The models work under the assumption that human operators use a threshold, a multiple of the steady-state standard deviation, on certain signals to detect transitions. The models developed for the transition from single to double integrator dynamics and vice versa are based on the tracking error and system output acceleration, respectively. They have an accuracy of 88.9\% and 99.4\%, respectively. However, the estimation of the detection lags remains a limitation of both models. Nonetheless, this research helped confirm the tracking error can be used in a model for the transition from single to double integrator dynamics, proposed a model for the opposite transition, and identified that the relationship between control inputs and the system's response is an important factor in the detection phase.

Future human-machine control tasks with preview (e.g., car driving) are expected to include automation for safety, but keep operators in charge for liability. Such shared control applications require time-varying human identification because the control feedback should be compatible with the operator's variable behavior. A promising time-domain identification algorithm is the Dual Extended Kalman Filter (DEKF), estimating human operator parameters from Van der El's preview model. In this article, the DEKF's time-varying identification performance is studied with realistic simulations, followed by human operator experiments in a fixed-base simulator. The investigation focuses on look-ahead time, indicating how much future information the operator uses for control. Compared to other parameters, look-ahead time is adapted most considerably with preview. The results suggest that this parameter should be initialized in a 0.25 s proximity of its actual value to make the DEKF converge within 30 s. Although only estimating look-ahead time while fixing the other parameters, the DEKF is capable of identifying time variations in preview. Based on the sigmoid results, the estimation bias increases linearly to 0.35 s at the largest 0.75 s steps in look-ahead time. For sine variations, the DEKF estimations are in phase with the look-ahead time until 0.03 rad/s. Between 0.03 rad/s and 0.4 rad/s the DEKF behaves as a lag function, and for higher frequencies the estimation response is decayed. For the first time, it is quantified how well the DEKF can identify variations in look-ahead time during preview tracking tasks. With further research, the DEKF might become capable of real-time identification, bringing the cybernetics community one step closer to intuitive shared control applications.

Haptic Shared Control (HSC) systems offer a means to naturally support human drivers in the transition to automated driving. Tuning of HSC requires knowledge of the neuromuscular system (NMS) of drivers. This MSc thesis project aimed to experimentally validate a linear parameter varying (LPV) technique for modeling driver neuromuscular admittance in time-varying conditions, scheduled with grip force. LPV estimations were compared with estimates made with a Recursive Least Squares (RLS) algorithm and, where possible, a linear time-invariant (LTI) estimation technique. A human-in-the loop experiment was performed in the HMILab fixed-base driving simulator at the AE faculty. To force drivers to change their neuromuscular admittance, a steering wheel manipulation task was designed, where drivers had to keep the steering wheel angle in between limits that were visually indicated on the dashboard, while a perturbation torque was applied. Changing the limits (narrow/wide) forced drivers to adjust their admittance. The results show that the relation between admittance and grip force is not only different between subjects, but also for the same drivers in time-invariant and time-varying conditions. This leads to fundamental scheduling issues for the LPV modelling approach. The variance accounted for (VAF) for the estimated steering wheel angle varied between 0% and 90%. To overcome these scheduling problems, individual LPV models can be constructed on time-varying experiment data. RLS estimates reproduced the system output data accurately with average VAF between 80 and 90% for all experiments. RLS low-frequency (< 2.5Hz) FRF estimates closely match the LTI FRF estimates. The largest variations in driver admittance occur in this lower frequency domain, making the RLS algorithm an effective tool for identifying changes in driver neuromuscular admittance, which can be applied online.

Parkinson’s Disease is a neurodegenerative disease that has a decline in motor behaviour as one of its main symptoms. This decline is currently monitored using subjective measures, such as questionnaires and clinical observations. More detailed and objective tracking of this decline can improve treatment of the disease and allow for earlier identification. Earlier work in this area has lead to the development of a proof-of-concept for detecting behavioural changes in motor performance, but the linear regression model used was limited in its accuracy and had a high latency. This paper uses a non-linear Least Squares Support Vector Regression (LS-SVR) model to increase performance in those areas. LS-SVR was chosen for its ability to regress complex non-linear relationships and because it is highly adaptable and scalable, yet easy to understand and work with. Test data were simulated based on earlier measured data which resulted in a complete and varied data set that allows for exploring many different situations.
Results from the machine learning model when applied to the test data are promising. With the right combination of hyperparameter settings an improvement of 80% in accuracy was reached, and latency could be reduced by 30%. Additionally, a sensitivity analysis of the hyperparameters revealed further room for improvement with more careful tuning. Finally, it was shown that by using LS-SVR to make predictions on data from future trials a further reduction in latency can be achieved. Overall, the new model shows definite improvement over the earlier model and can be developed further in subsequent steps towards a clinical applicable method.

Driving simulators have been used in the automotive industry for many years now. They have been vastly employed for conducting tests in a safe, reproducible and controlled immersive virtual environment. The ability of the simulator to recreate the in-vehicle experience for the occupant is established through motion cueing algorithms. Such algorithms have consistently been developed with model predictive control (MPC) acting as the main control technique. Currently available MPC based methods either compute the optimal controller online or derive an explicit control law in an offline setting. These approaches limit the applicability of MPC for real-time applications due to online computational expense and offline memory storage issues.

This thesis report presents a solution to deal with issues of offline and online solving through a combined/hybrid approach. For this, explicit MPC is used to provide an initial guess as warm start for the implicit MPC based motion cueing algorithm. From the simulations, it was observed that the presented hybrid approach was able to reduce online computational load by shifting it offline using the explicit MPC. Further, braking constraints and adaptive washout weights were implemented in the hybrid motion cueing algorithm, to improve the specific force tracking performance and reduce any false cue occurrences. Finally, emulator studies were performed to realize driving simulator performance with the hybrid MPC approach. The thesis concludes by showing the improvements made with the developed algorithm and proposes recommendations for future work.