It is our great pleasure to welcome you to the 26th International Conference on Multimodal Interaction (ICMI 2024) in San Jose, Costa Rica. This is the first ICMI held in Latin America. ICMI is the premier international forum for human-centered multimodal interaction and social artificial intelligence (AI). Multimodal human-centered AI includes machine learning and computational techniques, such as representation learning, fusion, data and systems. The study of social interactions includes both human-human and human-machine interactions. A unique aspect of ICMI is its multidisciplinary nature which values both scientific discoveries and technical modeling of human interaction towards positive and societally beneficial applications. We are keen to showcase the recent advances in the field to the community and provide an inclusive forum to develop and exchange new ideas. To this end, a new initiative committee has been established to organize events for newcomers and facilitate inclusion. [...]@en

Describing and analysing learner behaviour using sequential data and analysis is becoming more and more popular in Learning Analytics. Nevertheless, we found a variety of definitions of learning sequences, as well as choices regarding data aggregation and the methods implemented for analysis. Furthermore, sequences are used to study different educational settings and serve as a base for various interventions. In this literature review, the authors aim to generate an overview of these aspects to describe the current state of using sequence analysis in educational support and learning analytics. The 74 included articles were selected based on the criteria that they conduct empirical research on an educational environment using sequences of learning actions as the main focus of their analysis. The results enable us to highlight different learning tasks where sequences are analysed, identify data mapping strategies for different types of sequence actions, differentiate techniques based on purpose and scope, and identify educational interventions based on the outcomes of sequence analysis. @en

As social robots see increasing deployment within the general public, improving the interaction with those robots is essential. Spoken language offers an intuitive interface for the human–robot interaction (HRI), with dialogue management (DM) being a key component in those interactive systems. Yet, to overcome current challenges and manage smooth, informative, and engaging interaction, a more structural approach to combining HRI and DM is needed. In this systematic review, we analyze the current use of DM in HRI and focus on the type of dialogue manager used, its capabilities, evaluation methods, and the challenges specific to DM in HRI. We identify the challenges and current scientific frontier related to the DM approach, interaction domain, robot appearance, physical situatedness, and multimodality.@en

Although laughter is known to be a multimodal signal, it is primarily annotated from audio. It is unclear how laughter labels may differ when annotated from modalities like video, which capture body movements and are relevant in in-the-wild studies. In this work we ask whether annotations of laughter are congruent across modalities, and compare the effect that labeling modality has on machine learning model performance. We compare annotations and models for laughter detection, intensity estimation, and segmentation, using a challenging in-the-wild conversational dataset with a variety of camera angles, noise conditions and voices. Our study with 48 annotators revealed evidence for incongruity in the perception of laughter and its intensity between modalities, mainly due to lower recall in the video condition. Our machine learning experiments compared the performance of modern unimodal and multi-modal models for different combinations of input modalities, training, and testing label modalities. In addition to the same input modalities rated by annotators (audio and video), we trained models with body acceleration inputs, robust to cross-contamination, occlusion and perspective differences. Our results show that performance of models with body movement inputs does not suffer when trained with video-acquired labels, despite their lower inter-rater agreement.

@en

In ongoing and consecutive conversations with persons, a social robot has to determine which aspects to remember and how to address them in the conversation. In the health domain, important aspects concern the health-related goals, the experienced progress (expressed sentiment) and the ongoing motivation to pursue them. Despite the progress in speech technology and conversational agents, most social robots lack a memory for such experience sharing. This paper presents the design and evaluation of a conversational memory for personalized behavior change support conversations on healthy nutrition via memory-based motivational rephrasing. The main hypothesis is that referring to previous sessions improves motivation and goal attainment, particularly when references vary. In addition, the paper explores how far motivational rephrasing affects user's perception of the conversational agent (the virtual Furhat). An experiment with 79 participants was conducted via Zoom, consisting of three conversation sessions. The results showed a significant increase in participants' change in motivation when multiple references to previous sessions were provided.@en

This study investigates whether an agent-based Negotiation Training System (NTS) can teach women Strategic Empathy - a recently introduced negotiation strategy based on perspective taking - and whether this can improve their negotiation performance. Developed and tested through an interaction-based real-time experiment was a NTS that integrated instructions on how to utilize Strategic Empathy. Women in the experimental group showed significantly higher levels of perspective-taking compared to the control group, and their understanding and use of Strategic Empathy increased over time. Also, a significant positive effect was found of Strategic Empathy on women's self-efficacy. No significant positive effect was found of Strategic Empathy on persistence. The high cognitive load of the experiment and a lack of intrinsic motivation may have caused this finding. Overall, this work demonstrates the applicability of using NTS to teach Strategic Empathy, and its effectiveness for enhancing women's self-efficacy in salary negotiations. @en

When working in a group, it is essential to understand each other's viewpoints to increase group cohesion and meeting productivity. This can be challenging in teams: participants might be left misunderstood and the discussion could be going around in circles. To tackle this problem, previous research on group interactions has addressed topics such as dominance detection, group engagement, and group creativity. Conversational memory, however, remains a widely unexplored area in the field of multimodal analysis of group interaction. The ability to track what each participant or a group as a whole find memorable from each meeting would allow a system or agent to continuously optimise its strategy to help a team meet its goals. In the present paper, we therefore investigate what participants take away from each meeting and how it is reflected in group dynamics.As a first step toward such a system, we recorded a multimodal longitudinal meeting corpus (MEMO), which comprises a first-party annotation of what participants remember from a discussion and why they remember it. We investigated whether participants of group interactions encode what they remember non-verbally and whether we can use such non-verbal multimodal features to predict what groups are likely to remember automatically. We devise a coding scheme to cluster participants' memorisation reasons into higher-level constructs. We find that low-level multimodal cues, such as gaze and speaker activity, can predict conversational memorability. We also find that non-verbal signals can indicate when a memorable moment starts and ends. We could predict four levels of conversational memorability with an average accuracy of 44 %. We also showed that reasons related to participants' personal feelings and experiences are the most frequently mentioned grounds for remembering meeting segments.

@en

Listening to one another is essential to human-human interaction. In fact, we humans spend a substantial part of our day listening to other people, in private as well as in work settings. Attentive listening serves the function to gather information for oneself, but at the same time, it also signals to the speaker that he/she is being heard. To deduce whether our interlocutor is listening to us, we are relying on reading his/her nonverbal cues, very much like how we also use non-verbal cues to signal our attention. Such signaling becomes more complex when we move from dyadic to multi-party interactions. Understanding how humans use nonverbal cues in a multi-party listening context not only increases our understanding of human-human communication but also aids the development of successful human-robot interactions. This paper aims to bring together previous analyses of listener behavior analyses in human-human multi-party interaction and provide novel insights into gaze patterns between the listeners in particular. We are investigating whether the gaze patterns and feedback behavior, as observed in the human-human dialogue, are also beneficial for the perception of a robot in multi-party human-robot interaction. To answer this question, we are implementing an attentive listening system that generates multi-modal listening behavior based on our human-human analysis. We are comparing our system to a baseline system that does not differentiate between different listener types in its behavior generation. We are evaluating it in terms of the participant’s perception of the robot, his behavior as well as the perception of third-party observers.

@en

How human-like do conversational robots need to look to enable long-term human-robot conversation? One essential aspect of long-term interaction is a human's ability to adapt to the varying degrees of a conversational partner's engagement and emotions. Prosodically, this can be achieved through (dis)entrainment. While speech-synthesis has been a limiting factor for many years, restrictions in this regard are increasingly mitigated. These advancements now emphasise the importance of studying the effect of robot embodiment on human entrainment. In this study, we conducted a between-subjects online human-robot interaction experiment in an educational use-case scenario where a tutor was either embodied through a human or a robot face. 43 English-speaking participants took part in the study for whom we analysed the degree of acoustic-prosodic entrainment to the human or robot face, respectively. We found that the degree of subjective and objective perception of anthropomorphism positively correlates with acoustic-prosodic entrainment.

@en

We are organizing again the workshop on Interdisciplinary Insights into Group and Team Dynamics which is a joint effort between researchers in the the ICMI and INGRoup (Interdisciplinary Network for Group Research) communities. This workshop aims to provide a common destination for researchers to exchange ideas and collaborate. We have found in previous years that instigating interdisciplinary collaborations can be hard. The aim of this workshop is to sustain a joint community to foster continued cross-disciplinary exchange and mutual understanding.

@en

Context: Exploring the design space is an important process in a design task. In this study, we considered design space exploration for the learners in vocational education and training (VET). The goal of the study was to investigate how they explore the design space while focusing on the effect of a graph-like interface on the learner's understanding of the design space. With florists as the target profession, we investigated how the apprentices explore design variations, what they would gain from such activity, and how we can support this process. Approach: We developed a web application called BloomGraph that allows learners to explore design variations. It provides a graph-based interface that enables the systematic variation of design. Using the BloomGraph application, we conducted an experimental study with 44 florist apprentices in Switzerland to investigate the effect of the graph-based interface which provides a structured way of exploring the design space. The experimental group was given the graph-based interface to explore design variations while the control group had a linear-based interface. We compared them in terms of the number of bouquets explored, time of exploration, diversity of bouquets explored, and the learning gain in terms of the understanding of the design space measured using pre and post-Tests. We also analyzed the strategies adopted by the participants for the graph navigation and the visual exploration behavior using the eye gaze data. Findings: Our analysis shows that the graph-based interface fosters a better understanding of the size of the design space and more efficient navigation towards a goal design in terms of the number of intermediate designs but with longer exploration of each intermediate design compared to the linear-based interface. Regarding the behavioral patterns in graph exploration, the participants who showed more strategic behavior in the design choices acquired a better understanding of the design space. Additionally, we trained a model that predicts the next choice of a learner using eye tracking data. It provides a reasonable accuracy that opens new possibilities for future studies. Conclusion: The findings of this study support the feasibility of design space exploration as a digital activity for VET learners and show how the learners can benefit from it. The contribution of the paper includes the validation of the idea with florist apprentices and the demonstration of how the process can be supported using a structured interface and the learner behavior analysis. This paper shows how a design exploration activity can provide an added value in the learning of an apprentice in a design-related VET system.

@en

Visual expertise is a fundamental proficiency in many vocations and many questions have risen on the topic, with studies looking at experts and novices differences’ in observation (e.g., radiologists) or at ways to help novices achieve visual expertise (e.g., through annotations). However, most of these studies focus on white-collar professions and overlook vocational ones. For example, observing is uttermost important for fashion designers who spend most of their professional time on visual tasks related to creating patterns and garments or performing alterations. Therefore, this study focuses on trying to convey a professional way to look at images by exposing apprentices to images annotated (e.g., circles) by experts and identifying if their gaze (e.g., fixation durations and gaze coverage) and verbalisations (i.e., images descriptions) are affected. The study was conducted with 38 apprentices that were exposed to sequential sets of images depicting shirts, first non-annotated (pre-test), then annotated for the experimental group and non-annotated for the control group (training 1 and training 2), and finally non-annotated (post-test). Also, in the pre and post-test and in training 2 apprentices had to verbally describe each image. Gaze was recorded with the Tobii X2–60 tracker. Results for fixation durations showed that the experimental group looked longer in the annotated part of the shirt in training 1 and in the shirt’s central part at post-test. However, the experimental group did not cover a significantly larger area of the shirt compared to control and verbalisations show no difference between the groups at post-test.@en

There has been gathering momentum over the last 10 years in the study of group behavior in multimodal multiparty interactions. While many works in the computer science community focus on the analysis of individual or dyadic interactions, we believe that the study of groups adds an additional layer of complexity with respect to how humans cooperate and what outcomes can be achieved in these settings. Moreover, the development of technologies that can help to interpret and enhance group behaviours dynamically is still an emerging field. Social theories that accompany the study of groups dynamics are in their infancy and there is a need for more interdisciplinary dialogue between computer scientists and social scientists on this topic. This workshop has been organised to facilitate those discussions and strengthen the bonds between these overlapping research communities

@en

Virtual agents are increasingly being used for communication training such as public speaking-, job interviews-, as well as negotiation training. In these use-cases the agent is generally taking on the role of interviewer and its behaviour is altered according to the nonverbal cues of its human interlocutor. However, understanding how the agent's non-verbal cues influence human behaviour, perception or interactions outcomes is equally important. This contributes to appropriate behaviour generation in agents, but also to our understanding of the intricate interplay of non-verbal behaviours on human perception and interaction outcomes. This study focuses specifically on the perception of vocal dominance in human-agent negotiation. Earlier research showed that the perception of dominance influences decision making in the course of negotiations, as do concessions tactics. However, the effect of voice as well as the effect of the negotiator type in this regard have been so far under-explored. To close this gap, an online experiment was conducted, in which a total number of 121 participants negotiated with conversational agents displaying either low or high vocal dominance, and an individualistic or neutral concession tactic. The results showed that when taking into account the self-reported type of negotiator, significant differences caused by vocal dominance were evident in regard to the number of negotiation rounds and perceived fairness. The number of rounds was significantly higher for the competitive participant type negotiating with the low vocal dominance agent, and the perceived fairness was significantly lower with the collaborative participant type negotiating with the high vocal dominance agent.

@en

For the training of interpersonal skills, such as those required in the medical field, virtual agents can provide a safe environment for practice. However, many agent systems are not developed with the ability to understand non-verbal input. Being able to automatically parse such input is essential for the practice of interpersonal skills such as empathy. Currently, it is still an open question which prosodic or visual features would aid automatic classification of empathy and how this knowledge can be used to support the practice of these skills. As a first step towards this goal, we report on 42 second-year nursing students practicing their empathy skills with a virtual patient or through collaborative role playing. We found that across both the role playing and simulation, students assessed their empathy as increasing over time but as higher during the role playing. This work contributes to the continued development of virtual agents for the training of interpersonal skills.

@en

Engagement is a concept of the utmost importance in human-computer interaction, not only for informing the design and implementation of interfaces, but also for enabling more sophisticated interfaces capable of adapting to users. While the notion of engagement is actively being studied in a diverse set of domains, the term has been used to refer to a number of related, but different concepts. In fact it has been referred to across different disciplines under different names and with different connotations in mind. Therefore, it can be quite difficult to understand what the meaning of engagement is and how one study relates to another one accordingly. Engagement has been studied not only in human-human, but also in human-agent interactions i.e., interactions with physical robots and embodied virtual agents. In this overview article we focus on different factors involved in engagement studies, distinguishing especially between those studies that address task and social engagement, involve children and adults, are conducted in a lab or aimed for long term interaction. We also present models for detecting engagement and for generating multimodal behaviors to show engagement.

@en

Immersive virtual reality (IVR) offers possibilities of creating a learner-centric environment that can provide more presence and engagement for students leading to an enhanced learning experience compared to conventional classroom practices. However, the potential of IVR in vocational education and training (VET) has not yet been explored in-depth, and it is an open question of whether it can effectively support learner creation in a designing task. In this paper, we present an IVR application developed to support gardener apprentices in designing gardens. Using this application, we conducted an experimental study with gardener apprentices to investigate the effect of the IVR interface compared to paper sketching and learner behavior on the proportion, composition, and creativity of the design outcome. Additionally, we investigated how it can be combined with a paper sketching activity to improve its effectiveness. Our analysis shows that the IVR interface can be more effective for the proportion aspect, but this may be limited to students that are able to use it after working with paper. In terms of the combination order, the effectiveness of IVR on the design quality was improved when it was carried out after the paper sketching and this ordering produced a more effective outcome for the proportion and composition aspects. Finally, our results show that IVR design quality is related to learner behaviors such as the time spent on designing and the number of simulations used. This study demonstrates the effectiveness of IVR applications in supporting designing skills and how effectiveness can be improved by combining it with a conventional method of practice.

@en

We previously introduced a responsive joint attention system that uses multimodal information from users engaged in a spatial reasoning task with a robot and communicates joint attention via the robot's gaze behavior [25]. An initial evaluation of our system with adults showed it to improve users' perceptions of the robot's social presence. To investigate the repeatability of our prior findings across settings and populations, here we conducted two further studies employing the same gaze system with the same robot and task but in different contexts: Evaluation of the system with external observers and evaluation with children. The external observer study suggests that third-person perspectives over videos of gaze manipulations can be used either as a manipulation check before committing to costly real-time experiments or to further establish previous findings. However, the replication of our original adults study with children in school did not confirm the effectiveness of our gaze manipulation, suggesting that different interaction contexts can affect the generalizability of results in human-robot interaction gaze studies.

@en