Introduction: Augmented reality (AR) has been increasingly studied in transportation, particularly for drivers and pedestrians interacting with automated vehicles (AVs). Previous research evaluated AR interfaces using online video-based questionnaires but lacked human-subject research in immersive environments. This study examined if prior online evaluations of nine AR interfaces could be replicated in an immersive virtual environment and if AR interface effectiveness depends on pedestrian attention allocation. 

Methods: Thirty participants completed 120 trials in a CAVE-based simulator with yielding and non-yielding AVs, rating the interface’s intuitiveness and crossing the road when they felt safe. To emulate visual distraction, participants had to look into an attention-attractor circle that disappeared 1 s after the interface appeared. 

Results: The results showed that intuitiveness ratings from the current CAVE-based study and the previous online study correlated strongly (r ≈ 0.90). Head-locked interfaces and familiar designs (augmented traffic lights, zebra crossing) yielded higher intuitiveness ratings and quicker crossing initiations than vehicle-locked interfaces. Vehicle-locked interfaces were less effective when the attention-attractor was on the environment’s opposite side, while head-locked interfaces were relatively unaffected by attention-attractor position. 

Discussion: In conclusion, this ‘AR in VR’ study shows strong congruence between intuitiveness ratings in a CAVE-based study and online research, and demonstrates the importance of interface placement in relation to user gaze direction.

@en

Many collisions between pedestrians and cars are caused by poor visibility, such as occlusion by a parked vehicle. Augmented reality (AR) could help to prevent this problem, but it is unknown to what extent the augmented information needs to be embedded into the world. In this virtual reality experiment with a head-mounted display (HMD), 28 participants were exposed to AR designs, in a scenario where a vehicle approached from behind a parked vehicle. The experimental conditions included a head-locked live video feed of the occluded region, meaning it was fixed in a specific location within the view of the HMD (VideoHead), a world-locked video feed displayed across the street (VideoStreet), and two conformal diminished reality designs: a see-through display on the occluding vehicle (VideoSeeThrough) and a solution where the occluding vehicle has been made semi-transparent (TransparentVehicle). A Baseline condition without augmented information served as a reference. Additionally, the VideoHead and VideoStreet conditions were each tested with and without the addition of a guiding arrow indicating the location of the approaching vehicle. Participants performed 42 trials, 6 per condition, during which they had to hold a key when they felt safe to cross. The keypress percentages and responses from additional questionnaires showed that the diminished-reality TransparentVehicle and VideoSeeThrough designs came out most favourably, while the VideoHead solution caused some discomfort and dissatisfaction. An analysis of head yaw angle showed that VideoHead and VideoStreet caused divided attention between the screen and the approaching vehicle. The use of guiding arrows did not contribute demonstrable added value. AR designs with a high level of local embeddedness are beneficial for addressing occlusion problems when crossing. However, the head-locked solutions should not be immediately dismissed because, according to the literature, such solutions can serve tasks where a salient warning or instruction is beneficial.

Marie Skłodowska-Curie Actions; Innovative Training Networks (ITN); SHAPE-IT; Grant number 860410

Publication date: 27 May 2024

DOI: 10.1007/s10055-024-01017-9@en

Introduction: Communication from automated vehicles (AVs) to pedestrians using augmented reality (AR) could positively contribute to traffic safety. However, previous AR research for pedestrians was mainly conducted through online questionnaires or experiments in virtual environments instead of real ones. Methods: In this study, 28 participants conducted trials outdoors with an approaching AV and were supported by four different AR interfaces. The AR experience was created by having participants wear a Varjo XR-3 headset with see-through functionality, with the AV and AR elements virtually overlaid onto the real environment. The AR interfaces were vehicle-locked (Planes on vehicle), world-locked (Fixed pedestrian lights, Virtual fence), or head-locked (Pedestrian lights HUD). Participants had to hold down a button when they felt it was safe to cross, and their opinions were obtained through rating scales, interviews, and a questionnaire. Results: The results showed that participants had a subjective preference for AR interfaces over no AR interface. Furthermore, the Pedestrian lights HUD was more effective than no AR interface in a statistically significant manner, as it led to participants more frequently keeping the button pressed. The Fixed pedestrian lights scored lower than the other interfaces, presumably due to low saliency and the fact that participants had to visually identify both this AR interface and the AV. Discussion: In conclusion, while users favour AR in AV-pedestrian interactions over no AR, its effectiveness depends on design factors like location, visibility, and visual attention demands. In conclusion, this work provides important insights into the use of AR outdoors. The findings illustrate that, in these circumstances, a clear and easily interpretable AR interface is of key importance.

Marie Skłodowska-Curie Actions; Innovative Training Networks (ITN); SHAPE-IT; Grant number 860410

Publication date: 30 January 2024

DOI: 10.3389/frobt.2024.1324060

@en

This thesis explores how automated vehicles will interact with pedestrians in the urban environment through augmented reality technology. Nine distinct AR interfaces were designed, developed, and evaluated to assess how different design elements (symbols, text, colour) and distinct mappings of the AR (on the road, on the vehicle, or head-locked) would affect comprehension, and ultimately whether the pedestrian would trust and be convinced to cross in front of an automated vehicle displaying a safe message. Using increasing levels of ecological validity, from an online questionnaire to a CAVE simulator and an AR HMD experiment, the evaluation also explored how different AR anchoring (and mapping) positions affect pedestrians' crossing initiation times and the intuitiveness of the message. The thesis also explores the use of diminished reality (removal of information) to assist pedestrians in occluded scenarios, as well as the utilisation of Large Language Models in evaluating qualitative data in experiments. The outcomes of the thesis are a set of guidelines based on empirical evidence on how to design effective AR interfaces which promote safe and transparent interactions between pedestrians and automated vehicles.@en

While great strides have been taken in advancing the field of Human-Robot Interaction (HRI), challenges abound in understanding and improving how Autonomous Vehicles (AVs) will interact with and within society. Through this paper, the authors attempt to paint the picture of challenges unique to the study and advancement of interfaces between AVs and vulnerable road users (VRUs). In turn, these gaps in research highlight the opportunities for academia, industry, and public policy to collaborate and advance the state of the art of AV-VRU interaction, and the need for a dedicated forum for sharing insights across these various sectors.@en

Augmented Reality (AR) technology could be utilised to assist pedestrians in navigating safely through traffic. However, whether potential users would understand and use such AR solutions is currently unknown. Nine novel AR interfaces for pedestrian-vehicle communication, previously developed using an experience-based design method, were evaluated through an online questionnaire study completed by 992 respondents in Germany, the Netherlands, Norway, Sweden, and the United Kingdom. The AR indicated whether it was safe to cross the road in front of an approaching automated vehicle. Each interface was rated for its intuitiveness and convincingness, aesthetics, and usefulness. Moreover, comments were collected for qualitative analysis. The results indicated that interfaces that employed traditional design elements from existing traffic, and head-up displays, received the highest ratings overall. Statistical results also showed that there were no significant effects of country, age, and gender on interface acceptance. Thematic analysis of the textual comments offered detail on each interface design's stronger and weaker points, and revealed unintended effects of certain designs. In particular, some of the interfaces were commented on as being dangerous or scary, or were criticised that they could be misinterpreted in that they signal that something is wrong with the vehicle, or that they could occlude the view of the vehicle. The current findings highlight the limitations of experience-based design, and the importance of applying legacy design principles and involving target users in design and evaluation. Future research should be conducted in scenarios in which pedestrians actually interact with approaching vehicles.

Marie Skłodowska-Curie Actions; Innovative Training Networks (ITN); SHAPE-IT; Grant number 860410

Publication date: 21 February 2023

DOI: 10.1016/j.trf.2023.02.005

@en

ChatGPT could serve as a tool for text analysis within the field of Human-Computer Interaction, though its validity requires investigation. This study applied ChatGPT to: (1) textbox questionnaire responses on nine augmented-reality interfaces, (2) interview data from participants who experienced these interfaces in a virtual simulator, and (3) transcribed think-aloud data of participants who viewed a real painting and its replica. Using a hierarchical approach, ChatGPT produced scores or summaries of text batches, which were then aggregated. Results showed that (1) ChatGPT generated sentiment scores of the interfaces that correlated extremely strongly (r > 0.99) with human rating scale outcomes and with a rule-based sentiment analysis method (criterion validity). Additionally, (2) by inputting automatically transcribed interviews to ChatGPT, it provided meaningful meta-summaries of the qualities of the interfaces (face validity). One meta-summary analysed in depth was found to have substantial but imperfect overlap with a content analysis conducted by an independent researcher (criterion validity). Finally, (3) ChatGPT's summary of the think-aloud data highlighted subtle differences between the real painting and the replica (face validity), a distinction corresponding with a keyword analysis (criterion validity). In conclusion, our research indicates that, with appropriate precautions, ChatGPT can be used as a valid tool for analysing text data.

Marie Skłodowska-Curie Actions; Innovative Training Networks (ITN); SHAPE-IT; Grant number 860410

Publication date: 13 September 2023

DOI: 10.1098/rsos.231053

@en

While great strides have been taken in advancing the field of Human-Robot Interaction (HRI), challenges abound in understanding and improving how Autonomous Vehicles (AVs) will interact with and within society. Through this paper, the authors attempt to paint the picture of challenges unique to the study and advancement of interfaces between AVs and vulnerable road users (VRUs). In turn, these gaps in research highlight the opportunities for academia, industry, and public policy to collaborate and advance the state of the art of AV-VRU interaction, and the need for a dedicated forum for sharing insights across these various sectors.@en

This project aims to explore whether augmented reality (AR) technology is suitable for communication between automated vehicles (AVs) and vulnerable road users (VRUs). Several AR designs have been proposed, and user tested to measure factors such as interpretability, and trust, with increasing ecological validity.@en

Nine AR interfaces designed using an experience- based, and theoretically informed design approach, were presented in an online questionnaire for user evaluation.
Statistical analysis of presented measures, and the computation of an overall composite score revealed a preference towards traditional and familiar traffic elements.@en

This study explored how people look at The Night Watch (1642), Rembrandt's masterpiece. Twenty-one participants each stood in front of the painting for 5 min, while their eyes were recorded with a mobile eye-tracker and their thoughts were verbalized with a think-aloud method. We computed a heatmap of the participants’ attentional distribution using a novel markerless mapping method. The results showed that the participants’ attention was mainly directed at the faces of the two central figures, the bright mascot girl in the painting, and detailed elements such as the apparel of the key figures. The eye-movement analysis and think-aloud data also showed that participants’ attention shifted from the faces of the key figures to other elements of the scene over the course of the 5 min. Our analyses are consistent with the theory that Rembrandt used light and texture to capture the viewer's attention. Finally, the robustness of the eye-tracking method was demonstrated by replicating the study on a smaller replica.

@en

This project aims to explore whether augmented reality (AR) technology is suitable for communication between automated/autonomous vehicles (AVs) and vulnerable road users (VRUs). The issues of preferred design, interpretability, comparison to current communication strategies, and trust are or will be explored.@en

The future urban environment may consist of mixed traffic in which pedestrians interact with automated vehicles (AVs). However, it is still unclear how AVs should communicate their intentions to pedestrians. Augmented reality (AR) technology could transform the future of interactions between pedestrians and AVs by offering targeted and individualized communication. This paper presents nine prototypes of AR concepts for pedestrian-AV interaction that are implemented and demonstrated in a real crossing environment. Each concept was based on expert perspectives and designed using theoretically-informed brainstorming sessions. Prototypes were implemented in Unity MARS and subsequently tested on an unmarked road using a standalone iPad Pro with LiDAR functionality. Despite the limitations of the technology, this paper offers an indication of how future AR systems may support future pedestrian-AV interactions.

Marie Skłodowska-Curie Actions; Innovative Training Networks (ITN); SHAPE-IT; Grant number 860410

Publication date: 20 September 2021

DOI: 10.1145/3409118.3475149

@en

This Deliverable starts with a short overview of the design principles and guidelines developed for current Human Machine Interfaces (HMIs), which are predominantly developed for manually driven vehicles, or those with a number of Advanced Driver Assistance Systems (ADAS), at SAE Levels 0 and 1 (SAE, 2018). It then provides an overview of how the addition of more capable systems, and the move to higher levels of vehicle automation, is changing the role the human inside an Automated Vehicle (AV), and the ways in which future automated vehicles at higher levels of automation (SAE level 4 and 5) must communicate with other road users, in the absence of an “in charge” human driver.

It is argued that such changes in the role of the driver, and more transfer of control to the AV and its different functionalities, means that there will be more emphasis on the roles and responsibilities of HMIs for future AVs. In parallel, the multifaceted nature of these HMI, presented from different locations, both in and outside the vehicles, using a variety of modalities, and engaging drivers in a two-way interaction, means that a new set of design guidelines are required, to ensure that the humans interacting with AVs (inside and outside the vehicle) are not distracted and overloaded, that they remain situation aware and understand the capabilities and limitations of the system, having the right mental model of system capabilities and their responsibilities, as responsible road users, at all times

Following a summary of suggested frameworks and design principles which highlight the significant change needed for new AV HMIs, an overview of results from studies investigating human interaction with internal (or iHMIs), and external (or eHMIs), is provided, with examples of new and innovative methods of communication between humans and their vehicles.

The Deliverable then provides a summary of the innovative approaches that will be tackled by the ESRs of the project, which focus on factors such as use of AI and AR for future design of more intuitive and transparent HMI, studying how HMI can support the long term interaction of humans with AVs, and the use of neuroergonomic methods for developing safer HMIs. The Deliverable concludes by summarising how each ESR’s project contributes to the development of HMIs for future AVs.@en

Automated driving research over the past decades has mostly focused on highway environments. Recent technological developments have drawn researchers and manufacturers to look ahead at introducing automated driving in cities. The current position paper examines this challenge from the viewpoint of scientific experts. Sixteen Human Factors researchers were interviewed about their personal perspectives on automated vehicles (AVs) and the interaction with VRUs in the future urban environment. Aspects such as smart infrastructure, external human-machine interfaces (eHMIs), and the potential of augmented reality (AR) were addressed during the interviews. The interviews showed that the researchers believed that fully autonomous vehicles will not be introduced in the coming decades and that intermediate levels of automation, specific AV services, or shared control will be used instead. The researchers foresaw a large role of smart infrastructure and expressed a need for AV-VRU segregation, but were concerned about corresponding costs and maintenance requirements. The majority indicated that eHMIs will enhance future AV-VRU interaction, but they noted that implicit communication will remain dominant and advised against text-based and instructive eHMIs. AR was commended for its potential in assisting VRUs, but given the technological challenges, its use, for the time being, was believed to be limited to scientific experiments. The present expert perspectives may be instrumental to various stakeholders and researchers concerned with the relationship between VRUs and AVs in future urban traffic.

Marie Skłodowska-Curie Actions; Innovative Training Networks (ITN); SHAPE-IT; Grant number 860410

Publication date: 4 January 2021

DOI: 10.1016/j.trip.2020.100293

@en

The progress in technology development over the past decades, both with respect to software and hardware, offers the vision of automated vehicles as means of achieving zero fatalities in traffic. However, the promises of this new technology – an increase in road safety, traffic efficiency, and user comfort – can only be realized if this technology is smoothly introduced into the existing traffic system with all its complexities, constraints, and requirements. SHAPE- IT will contribute to this major undertaking by addressing research questions relevant for the development and introduction of automated vehicles in urban traffic scenarios. Previous research has pointed out several research areas that need more attention for a successful implementation and deployment of human-centred vehicle automation in urban environments.

In SHAPE-IT, for example, a better understanding of human behaviour and the underlying psychological mechanisms will lead to improved models of human behaviour that can help to predict the effects of automated systems on human behaviour already during system development. Such models can also be integrated into the algorithms of automated vehicles, enabling them to better understand the human interaction partners’ behaviours.

Further, the development of vehicle automation is much about technology (software and hardware), but the users will be humans and they will interact with humans both inside and outside of the vehicle. To be successful in the development of automated vehicles functionalities, research must be performed on a variety of aspects. Actually, a highly interdisciplinary team of researchers, bringing together expertise and background from various scientific fields related to traffic safety, human factors, human-machine interaction design and evaluation, automation, computational modelling, and artificial intelligence, is likely needed to consider the human-technology aspects of vehicle automation.

Accordingly, SHAPE-IT has recruited fifteen PhD candidates (Early Stage Researchers – ESRs), that work together to facilitate this integration of automated vehicles into complex urban traffic by performing research to support the development of transparent, cooperative, accepted, trustworthy, and safe automated vehicles. With their (and their supervisors’) different scientific background, the candidates bring different theoretical concepts and methodological approaches to the project. This interdisciplinarity of the project team offers the unique possibility for each PhD candidate to address research questions from a broad perspective – including theories and methodological approaches of other interrelated disciplines. This is the main reason why SHAPE-IT has been funded by the European Commission’s Marie Skłodowska-Curie Innovative Training Network (ITN) program that is aimed to train early state researchers in multidisciplinary aspects of research including transferable skills. With the unique scope of SHAPE-IT, including the human-vehicle perspective, considering different road-users (inside and outside of the vehicle), addressing for example trust, transparency, and safety, and including a wide range of methodological approaches, the project members can substantially contribute to the development and deployment of safe and appreciated vehicle automation in the cities of the future.

To achieve the goal of interdisciplinary research, it is necessary to provide the individual PhD candidate with a starting point, especially on the different and diverse methodological approaches of the different disciplines. The empirical, user-centred approach for the development and evaluation of innovative automated vehicle concepts is central to SHAPE- IT. This deliverable (D1.1 “Methodological Framework for Modelling and Empirical Approaches”) provides this starting point. That is, this document provides a broad overview of approaches and methodologies used and developed by the SHAPE-IT ESRs during their research. The SHAPE-IT PhD candidates, as well as other researchers and developers outside of SHAPE-IT, can use this document when searching for appropriate methodological approaches, or simply get a brief overview of research methodologies often employed in automated vehicle research.

The first chapter of the deliverable shortly describes the major methodological approaches to collect data relevant for investigating road user behaviour. Each subchapter describes one approach, ranging from naturalistic driving studies to controlled experiments in driving simulators, with the goal to provide the unfamiliar reader with a broad overview of the approach, including its scope, the type of data collected, and its limitations. Each subchapter ends with recommendations for further reading – literature that provide much more detail and examples.

The second chapter explains four different highly relevant tools for data collection, such as interviews, questionnaires, physiological measures, and as other current tools (the Wizard of Oz paradigm and Augmented and Virtual Reality). As in the first chapter this chapter provides the reader with information about advantages and disadvantages of the different tools and with proposed further readings.

The third chapter deals with computational models of human/agent interaction and presents in four subchapters different modelling approaches, ranging from models based on psychological mechanisms, rule-based and artificial intelligence models to simulation models of traffic interaction.

The fourth chapter is devoted to Requirements Engineering and the challenge of communicating knowledge (e.g., human factors) to developers of automated vehicles. When forming the SHAPE-IT proposal it was identified that there is a lack of communication of human factors knowledge about the highly technical development of automated vehicles. This is why it is highly important that the SHAPE-IT ESRs get training in requirement engineering. Regardless of the ESRs working in academia or industry after their studies it is important to learn how to communicate and disseminate the findings to engineers.

The deliverable ends with the chapter “Method Champions”. Here the expertise and association of the different PhD candidates with the different topics are made explicit to facilitate and encourage networking between PhDs with special expertise and those seeking support, especially with regards to methodological questions.@en

Throughout the past decades, art museums have employed a number of digital technology aids in order to enhance the visitor’s experience and memory retainment. Augmented reality has proven to be among the most successful applications in this regard as studies have shown it to be the most effective in eliciting the interest of the visitors and enhance their learning. Most of these augmented reality systems have been implemented on tablet computers or hand-held mobile devices. This chapter explores the effectiveness of such systems and presents an experiment conducted using an augmented reality system operating on a head mounted display. Gathered results and user evaluation indicated that the system offered an enhanced museum experience and improved the users’ concentration on exhibition, their art appreciation skills and more importantly elicited the visitor with a renewed interest in the artworks.

@en

Segmentation of anatomical imagery is important in several areas, such as forensics, medical analysis and educational material. The manual segmentation of such images and the subsequent labelling of regions is a very laborious task. We propose an interactive segmentation scheme which we evaluate on a new data set of anatomical imagery. We use a morphological tree-based segmentation method, known as the alpha-tree, together with a Hu-moment thresholding mechanism in order to extract segments from a number of structures. Both qualitative and quantitative results in anatomical imagery of embalmed head, arm and leg specimens indicate that the proposed method can produce meaningful segmentation outputs, which could facilitate further refined labelling.

@en

We present a system1 that detects faces in various paintings and subsequently recognise and points out any similarities that a certain face in one painting may have to another on a different artwork. The results would be ranked up according to similarity in a bid to produce an output that may assist art researchers to discover new links between different works which pertain to the same or different artist. Through various tests conducted, we have proved that our method was successful in exposing new links of similarity in various scenarios including cases where the human visual system failed to pinpoint any.@en