Appropriate Trust in Artificial Intelligence (AI) systems has rapidly become an important area of focus for both researchers and practitioners. Various approaches have been used to achieve it, such as confidence scores, explanations, trustworthiness cues, or uncertainty communication. However, a comprehensive understanding of the field is lacking due to the diversity of perspectives arising from various backgrounds that influence it and the lack of a single definition for appropriate trust. To investigate this topic, this paper presents a systematic review to identify current practices in building appropriate trust, different ways to measure it, types of tasks used, and potential challenges associated with it. We also propose a Belief, Intentions, and Actions (BIA) mapping to study commonalities and differences in the concepts related to appropriate trust by (a) describing the existing disagreements on defining appropriate trust, and (b) providing an overview of the concepts and definitions related to appropriate trust in AI from the existing literature. Finally, the challenges identified in studying appropriate trust are discussed, and observations are summarized as current trends, potential gaps, and research opportunities for future work. Overall, the paper provides insights into the complex concept of appropriate trust in human-AI interaction and presents research opportunities to advance our understanding on this topic.@en

Appropriate trust is an important component of the interaction between people and AI systems, in that "inappropriate"trust can cause disuse, misuse, or abuse of AI. To foster appropriate trust in AI, we need to understand how AI systems can elicit appropriate levels of trust from their users. Out of the aspects that influence trust, this article focuses on the effect of showing integrity. In particular, this article presents a study of how different integrity-based explanations made by an AI agent affect the appropriateness of trust of a human in that agent. To explore this, (1) we provide a formal definition to measure appropriate trust, (2) present a between-subject user study with 160 participants who collaborated with an AI agent in such a task. In the study, the AI agent assisted its human partner in estimating calories on a food plate by expressing its integrity through explanations focusing on either honesty, transparency, or fairness. Our results show that (a) an agent who displays its integrity by being explicit about potential biases in data or algorithms achieved appropriate trust more often compared to being honest about capability or transparent about the decision-making process, and (b) subjective trust builds up and recovers better with honesty-like integrity explanations. Our results contribute to the design of agent-based AI systems that guide humans to appropriately trust them, a formal method to measure appropriate trust, and how to support humans in calibrating their trust in AI.

@en

As human-machine teams become a more common scenario, we need to ensure mutual trust between humans and machines. More important than having trust, we need all teammates to trust each other appropriately. This means that they should not overtrust or undertrust each other, avoiding risks and inefficiencies, respectively. We usually think of natural trust, that is, humans trusting machines, but we should also consider artificial trust, that is, artificial agents trusting humans. Appropriate artificial trust allows the agents to interpret human behavior and predict their behavior in a certain context. In this chapter, we explore how we can define this context in terms of task and team characteristics. We present a taxonomy that shows how trust is context-dependent. In fact, we propose that no trust model presented in the literature fits all contexts and argue that our taxonomy facilitates the choice of the trust model that better fits a certain context. The taxonomy helps to understand which internal characteristics of the teammate (krypta) are important to consider and how they will show in behavior cues (manifesta). This taxonomy can also be used to help human-machine teams’ researchers in the problem definition and process of experimental design as it allows a detailed characterization of the task and team configuration. Furthermore, we propose a formalization of the belief of trust as context-dependent trustworthiness, and show how beliefs of trust can be used to reach appropriate trust. Our work provides a starting point to implement mutual appropriate trust in human-machine teams.@en

Appropriate trust, trust which aligns with system trustworthiness, in Artificial Intelligence (AI) systems has become an important area of research. However, there remains debate in the community about how to design for appropriate trust. This debate is a result of the complex nature of trust in AI, which can be difficult to understand and evaluate, as well as the lack of holistic approaches to trust. In this paper, we aim to clarify some of this debate by operationalising appropriate trust within the context of the Human-Centred AI Design (HCD) process. To do so, we organised three workshops with 13 participants total from design and development backgrounds. We carried out design activities to stimulate discussion on appropriate trust in the HCD process. This paper aims to help researchers and practitioners understand appropriate trust in AI through a design lens by illustrating how it interacts with the HCD process.@en

Intelligent systems are increasingly entering the workplace, gradually moving away from technologies supporting work processes to artificially intelligent (AI) agents becoming team members. Therefore, a deep understanding of effective human-AI collaboration within the team context is required. Both psychology and computer science literature emphasize the importance of trust when humans interact either with human team members or AI agents. However, empirical work and theoretical models that combine these research fields and define team trust in human-AI teams are scarce. Furthermore, they often lack to integrate central aspects, such as the multilevel nature of team trust and the role of AI agents as team members. Building on an integration of current literature on trust in human-AI teaming across different research fields, we propose a multidisciplinary framework of team trust in human-AI teams. The framework highlights different trust relationships that exist within human-AI teams and acknowledges the multilevel nature of team trust. We discuss the framework’s potential for human-AI teaming research and for the design and implementation of trustworthy AI team members.@en

Establishing an appropriate level of trust between people and AI systems is crucial to avoid the misuse, disuse, or abuse of AI. Understanding how AI systems can generate appropriate levels of trust among users is necessary to achieve this goal. This study focuses on the impact of displaying integrity, which is one of the factors that influence trust. The study analyzes how different integrity-based explanations provided by an AI agent affect a human’s appropriate level of trust in the agent. To explore this, we conducted a between-subject user study involving 160 participants who collaborated with an AI agent to estimate calories on a food plate, with the AI agent expressing its integrity in different ways through explanations. The preliminary results demonstrate that an AI agent that explicitly acknowledges honesty in its decision making process elicit higher subjective trust than those that are transparent about their decision-making process or fair about biases. These findings can aid in designing agent-based AI systems that foster appropriate trust from humans.@en

Social interactions are multisensory experiences. However, it is not well understood how technology-mediated smell can support social interactions, especially in collaborative tasks. To explore its effect on collaboration, we asked eleven pairs of users to work together on a writing task while wearing an interactive jewellery designed to emit scent in a controlled fashion. In a within-subjects experiment, participants were asked to collaboratively write a story about a standardized visual stimulus while exposed to with scent and without scent conditions. We analyzed video recordings and written stories using a combination of methods from HCI, psychology, sociology, and human communication research. We observed differences in both participants’ communication and creation of insightful stories in the with scent condition. Furthermore, scent helped participants recover from communication breakdown even though they were unaware of it. We discuss the possible implications of our findings and the potential of technology-mediated scent for collaborative activities.@en

As AI systems are increasingly involved in decision making, it also becomes important that they elicit appropriate levels of trust from their users. To achieve this, it is first important to understand which factors influence trust in AI. We identify that a research gap exists regarding the role of personal values in trust in AI. Therefore, this paper studies how human and agent Value Similarity (VS) influences a human's trust in that agent. To explore this, 89 participants teamed up with five different agents, which were designed with varying levels of value similarity to that of the participants. In a within-subjects, scenario-based experiment, agents gave suggestions on what to do when entering the building to save a hostage. We analyzed the agent's scores on subjective value similarity, trust and qualitative data from open-ended questions. Our results show that agents rated as having more similar values also scored higher on trust, indicating a positive effect between the two. With this result, we add to the existing understanding of human-agent trust by providing insight into the role of value-similarity.@en

Trust is an important element of any interaction, but especially when we are interacting with a piece of technology which does not think like we do. Therefore, AI systems need to understand how humans trust them, and what to do to promote appropriate trust. The aim of this research is to study trust through both a formal and social lens. We will be working on formal models of trust, but with a focus on the social nature of trust in order to represent how humans trust AI. We will then employ methods from human computer interaction research to study if these models work in practice, and what would eventually be necessary for systems to elicit appropriate levels of trust from their users. The context of this research will be AI agents which interact with their users to offer personal support.@en

In human-agent teams, how one teammate trusts another teammate should correspond to the latter's actual trustworthiness, creating what we would call appropriate mutual trust. Although this sounds obvious, the notion of appropriate mutual trust for human-agent teamwork lacks a formal definition. In this article, we propose a formalization which represents trust as a belief about trustworthiness. Then, we address mutual trust, and pose that agents can use beliefs about trustworthiness to represent how they trust their human teammates, as well as to reason about how their human teammates trust them. This gives us a formalization with nested beliefs about beliefs of trustworthiness. Next, we highlight that mutual trust should also be appropriate, where we define appropriate trust in an agent as the trust which corresponds directly to that agent's trustworthiness. Finally, we explore how agents can define their own trustworthiness, using the concepts of ability, benevolence and integrity. This formalization of appropriate mutual trust can form the base for developing agents which can promote such trust.@en

Mutual understanding via sharing and interpreting inner states is socially rewarding. Prior research shows that people find Brain-Computer Interfaces (BCIs) a suitable tool to implicitly communicate their cognitive states. In this paper, we conduct an online survey (N=43) to identify design parameters for systems that implicitly share cognitive states. We achieve this by designing a research probe called "SpotlessMind" to artistically share brain occupancy with another while considering the bystanders' experience to elicit user responses. Our results show that 98% would like to see the installation. People would use it as a gesture of openness and as a communication mediator. Abstracting visual, auditory, and somatosensory depictions is a good trade-off between understandability and users' privacy protection. Our work supports designing engaging prototypes that promote empathy, cognitive awareness and convergence between individuals.@en

Traditional television remote control presents frequent challenges to older adults. These challenges arise due to lack of feedback and poor design features such as labeling, size, spatial proximity, physical feel, etc. This paper describes the design of an accessible TV remote control (Potmote) created by employing potentiometers with Arduino to enhance tactile feedback and ease of channel selection with ergonomic controls. An experimental study was conducted with 15 older adults to understand how to design a system that would allow them to change channel numbers and volume levels. The result of experiment have shown positive feedback by the subjects.@en

There are numerous studies which suggest that perhaps music is truly the language of emotions. Music seems to have an almost willful, evasive quality, defying simple explanation, and indeed requires deeper neurophysiological investigations to gain a better understanding. The current study makes an attempt in that direction to explore the effect of context on music perception. To investigate the same, we measured Galvanic Skin Responses (GSR) and self-reported emotion on 18 participants while listening to different Ragas (musical stimulus) composed of different Rasa's (emotional expression) in the different context (Neutral, Pleasant, and Unpleasant). The IAPS pictures were used to induce the emotional context in participants. Our results from this study suggest that the context can modulate emotional response in music perception but only for a shorter time scale. Interestingly, here we demonstrate by combining GSR and self-reports that this effect gradually vanishes over time and shows emotional adaptation irrespective of context. The overall findings suggest that specific context effects of music perception are transitory in nature and gets saturated on a longer time scale.@en

Populations in developed societies show an increasingly higher life expectancy across the globe. To support older adults to live longer and healthier lives in the familiar surroundings of their homes, technological developments, such as robots and avatars, have a great potential. To investigate long-term interactions between older adults and a "bi-bodied conversational agent" (an agent that has both an avatar and a robot embodiment), a user-centred design approach was employed in the design and development of a conversational agent. Firstly, the requirements of the agent were elicited through a set of focus groups with the target users – older adults. Then, the agent was iteratively designed and implemented: a robot body and avatar body were created. Finally, a Wizard-of-Oz control panel was created to control and compare each of the two bodies. Current research outcomes describe the elicited requirements baseline of a bi-bodied conversational agent for older adults. Future research involves the use of this set-up to investigate long-term interaction between older adults and a bibodied conversational agent.@en

This paper describes the design and development of an embodied conversational agent (ECA) that provides a social interface for older adults. Following a user-centred design approach, we implemented a multimodal agent consisting of a virtual character and a robot. This so-called "bi-bodied conversational agent for elderly" was iteratively refined and developed through participatory design and rapid prototyping in 3 consecutive focus groups with a total of 21 elderly users. In addition to the two bodies, a Wizard-of-Oz control panel was developed, enabling researchers to control both bodies so as to respond to the user's instructions, questions, and remarks. The research resulted in a platform that can be used for future research on elderly-robot and elderly-avatar interaction. In addition, the research resulted in insights about elderly users' preferences regarding the appearance and design of a virtual and a robotic ECA (Embodied Conversational Agent), described in results that can be reused in future experiments involving ECA for elderly users.@en

There is a buzz around the world about building smart cities and smart homes. Smart cities in brief can be defined as a city which uses digital technologies or information-communication technologies to make life of living beings more efficient and comfortable. Smart Home, comprising smart devices in the home context, promises enormous possibilities to our future life. At the same time, it might have its own influence to change our living habits. This paper sets to provide a way-out on how can a city and a home aspiring to become smart can really achieve it with putting up challenges that will be encountered in the path of achieving the goal.@en