Existing research in measuring and mitigating gender bias predominantly centers on English, overlooking the intricate challenges posed by non-English languages and the Global South. This paper presents the first comprehensive study delving into the nuanced landscape of gender bias in Hindi, the third most spoken language globally. Our study employs diverse mining techniques, computational models, field studies and sheds light on the limitations of current methodologies. Given the challenges faced with mining gender biased statements in Hindi using existing methods, we conducted field studies to bootstrap the collection of such sentences. Through field studies involving rural and low-income community women, we uncover diverse perceptions of gender bias, underscoring the necessity for context-specific approaches. This paper advocates for a community-centric research design, amplifying voices often marginalized in previous studies. Our findings not only contribute to the understanding of gender bias in Hindi but also establish a foundation for further exploration of Indic languages. By exploring the intricacies of this understudied context, we call for thoughtful engagement with gender bias, promoting inclusivity and equity in linguistic and cultural contexts beyond the Global North.

@en

While existing literature has explored and revealed several insights pertaining to the role of human factors (e.g., prior experience, domain knowledge) and attributes of AI systems (e.g., accuracy, trustworthiness), there is a limited understanding around how the important task characteristics of complexity and uncertainty shape human decision-making and human-AI team performance. In this work, we aim to address this research and empirical gap by systematically exploring how task complexity and uncertainty infuence human-AI decision-making. Task complexity refers to the load of information associated with a task, while task uncertainty refers to the level of unpredictability associated with the outcome of a task. We conducted a between-subjects user study (N = 258) in the context of a trip-planning task to investigate the impact of task complexity and uncertainty on human trust and reliance on AI systems. Our results revealed that task complexity and uncertainty have a signifcant impact on user reliance on AI systems. When presented with complex and uncertain tasks, users tended to rely more on AI systems while demonstrating lower levels of appropriate reliance compared to tasks that were less complex and uncertain. In contrast, we found that user trust in the AI systems was not infuenced by task complexity and uncertainty. Our fndings can help inform the future design of empirical studies exploring human-AI decision-making. Insights from this work can inform the design of AI systems and interventions that are better aligned with the challenges posed by complex and uncertain tasks. Finally, the lens of diagnostic versus prognostic tasks can inspire the operationalization of uncertainty in human-AI decision-making studies.

@en

Conversational User Interfaces (CUIs) have been argued to have advantages over traditional GUIs due to having a more human-like interaction. The growing popularity of conversational agents has enabled humans to interact with machines more naturally. People are increasingly familiar with conversational interactions mediated by technology due to the widespread use of mobile devices and messaging services and a hungry market for conversational agents. Based on the recent advances in conversational AI, due to the proliferation of large language models, there are clear signs that the future of human-computer interaction will have a significant conversational component. Today, over two-thirds of the population on our planet has access to the Internet, with ever-lowering barriers to accessibility. This tutorial will showcase the benefits of employing novel conversational interfaces for crowd computing, human-AI decision making, health and well-being, and information retrieval. Given the widespread adoption of AI systems across several domains, we will discuss the potential of conversational interfaces in facilitating and mediating people's interactions with AI systems and the opportunities and challenges that lie at this intersection from the user modeling and personalization standpoint. The tutorial will include interactive elements and discussions and provide participants with practical insights to inform the design of effective conversational interfaces.

@en

Human intelligence continues to be essential in building ground-truth data, training sets, and for evaluating a plethora of systems. The democratized and distributed nature of online crowd work - an attractive and accessible feature that has led to the proliferation of the paradigm - has also meant that crowd workers may not always feel connected to their remote peers. Despite the prevalence of collaborative crowdsourcing practices, workers on many microtask crowdsourcing platforms work on tasks individually and are seldom directly exposed to other crowd workers. In this context, improving worker engagement on microtask crowdsourcing platforms is an unsolved challenge. At the same time, fostering a sense of community among workers can improve the sustainability and working conditions in crowd work. This work aims to increase worker engagement in conversational microtask crowdsourcing by leveraging evolving avatars that workers can customize as they progress through monotonous task batches. We also aim to improve group identifcation in individual tasks by creating a community space where workers can share their avatars and feelings on task completion. To this end, we carried out a preregistered between-subjects controlled study (N = 680) spanning fve experimental conditions and two task types. We found that evolving and customizable worker avatars can increase worker retention. The prospect of sharing worker avatars and task-related feelings in a community space did not consistently afect group identifcation. Our exploratory analysis indicated that workers who identify themselves as crowd workers experienced greater intrinsic motivation, subjective engagement, and perceived workload. Furthermore, we discuss how task diferences shape the relative efectiveness of our interventions. Our fndings have important theoretical and practical implications for designing conversational crowdsourcing tasks and in shaping new directions for research to improve crowd worker experiences.

@en

Web search has evolved into a platform people rely on for opinion formation on debated topics. Yet, pursuing this search intent can carry serious consequences for individuals and society and involves a high risk of biases. We argue that web search can and should empower users to form opinions responsibly and that the information retrieval community is uniquely positioned to lead interdisciplinary efforts to this end. Building on digital humanism---a perspective focused on shaping technology to align with human values and needs---and through an extensive interdisciplinary literature review, we identify challenges and research opportunities that focus on the searcher, search engine, and their complex interplay. We outline a research agenda that provides a foundation for research efforts toward addressing these challenges.@en

Powerful predictive AI systems have demonstrated great potential in augmenting human decision making. Recent empirical work has argued that the vision for optimal human-AI collaboration requires ‘appropriate reliance’ on AI systems. However, accurately estimating the trustworthiness of AI advice at the instance level is quite challenging, especially in the absence of performance feedback pertaining to the AI system. In practice, the performance disparity of machine learning models on out-of-distribution data makes the dataset-specific performance feedback unreliable in human-AI collaboration. Inspired by existing literature on critical thinking and mindsets, we propose debugging an AI system as an intervention to foster appropriate reliance. This paper explores whether a critical evaluation of AI performance within a debugging setting can better calibrate users’ assessment of an AI system. Through a quantitative empirical study (N = 234), we found that our proposed debugging intervention does not work as expected in facilitating appropriate reliance. Instead, we observe a decrease in reliance on the AI system — potentially resulting from an early exposure to the AI system’s weakness. Our findings have important implications for designing effective interventions to facilitate appropriate reliance and better human-AI collaboration.@en

Pilot studies are an essential cornerstone of the design of crowdsourcing campaigns, yet they are often only mentioned in passing in the scholarly literature. A lack of details surrounding pilot studies in crowdsourcing research hinders the replication of studies and the reproduction of findings, stalling potential scientific advances. We conducted a systematic literature review on the current state of pilot study reporting at the intersection of crowdsourcing and HCI research. Our review of ten years of literature included 171 articles published in the proceedings of the Conference on Human Computation and Crowdsourcing (AAAI HCOMP) and the ACM Digital Library. We found that pilot studies in crowdsourcing research (i.e., crowd pilot studies) are often under-reported in the literature. Important details, such as the number of workers and rewards to workers, are often not reported. On the basis of our findings, we reflect on the current state of practice and formulate a set of best practice guidelines for reporting crowd pilot studies in crowdsourcing research. We also provide implications for the design of crowdsourcing platforms and make practical suggestions for supporting crowd pilot study reporting.@en

Empirical studies have extensively investigated human decision-making processes in various domains where AI systems are incorporated. However, comparing and replicating these studies can be challenging due to different experimental configurations. Moreover, the existing contexts often have limited scope and may not fully capture the complexity of real-world decision-making scenarios that are riddled with varying levels of uncertainty. Our framework addresses these practical gaps by providing a configurable and reproducible environment for conducting human-AI decision-making studies in the route planning domain that captures many complexities of real-world scenarios. Researchers can customize parameters, conditions, and factors involved in decision-making tasks to help address research and empirical gaps through rigorous experiments. With various modules such as map generation, chat components, and different AI systems available within the "DecisionTime"framework, researchers can effortlessly design experiments exploring multiple aspects of human-AI interaction and decision-making.

@en

With the integration of AI systems into our daily lives, human-AI collaboration has become increasingly prevalent. Prior work in this realm has primarily explored the effectiveness and performance of individual human and AI systems in collaborative tasks. While much of decision-making occurs within human peers and groups in the real world, there is a limited understanding of how they collaborate with AI systems. One of the key predictors of human-AI collaboration is the characteristics of the task at hand. Understanding the influence of task characteristics on human-AI collaboration is crucial for enhancing team performance and developing effective strategies for collaboration. Addressing a research and empirical gap, we seek to explore how the features of a task impact decision-making within human-AI group settings. In a 2 × 2 between-subjects study (N = 256) we examine the effects of task complexity and uncertainty on group performance and behaviour. The participants were grouped into pairs and assigned to one of four experimental conditions characterized by varying degrees of complexity and uncertainty. We found that high task complexity and high task uncertainty can negatively impact the performance of human-AI groups, leading to decreased group accuracy and increased disagreement with the AI system. We found that higher task complexity led to a higher efficiency in decision-making, while a higher task uncertainty had a negative impact on efficiency. Our findings highlight the importance of considering task characteristics when designing human-AI collaborative systems, as well as the future design of empirical studies exploring human-AI collaboration.@en

When using web search engines to conduct inquiries on debated topics, searchers' interactions with search results are commonly affected by a combination of searcher and system biases. While prior work has mainly investigated these biases in isolation, there is a lack of a comprehensive understanding of web search on debated topics. Addressing this gap, we conducted an exploratory user study (N = 255), aimed at advancing the understanding of the intricate searcher-system interplay. Particularly, we investigated the relations between (i) search system exposure, searchers' attitude strength, prior knowledge, and receptiveness to opposing views, (ii) search interactions, and (iii) post-search epistemic states. We observed that search interaction was shaped by search system exposure, attitude strength, and prior knowledge, and that attitude change was influenced by the level of confirmation bias and initial attitude strength, but not search system exposure. Insights from this work suggest the need to adapt interventions that mitigate the risks of searcher and system bias to searchers' nuanced pre-search epistemic states. They further emphasize the threat of customizing the search ranking to enhance user satisfaction in the context of debated topics to responsible opinion formation.

@en

The ability to make appropriate delegation decisions is an important prerequisite of effective human-AI collaboration. Recent work, however, has shown that people struggle to evaluate AI systems in the presence of forecasting errors, falling well short of relying on AI systems appropriately. We use a pre-registered crowdsourcing study (N = 611) to extend this literature by two underexplored crucial features of human AI decision-making: choice independence and error type. Subjects in our study repeatedly complete two prediction tasks and choose which predictions they want to delegate to an AI system. For one task, subjects receive a decision heuristic that allows them to make informed and relatively accurate predictions. The second task is substantially harder to solve, and subjects must come up with their own decision rule. We systematically vary the AI system's performance such that it either provides the best possible prediction for both tasks or only for one of the two. Our results demonstrate that people systematically violate choice independence by taking the AI's performance in an unrelated second task into account. Humans who delegate predictions to a superior AI in their own expertise domain significantly reduce appropriate reliance when the model makes systematic errors in a complementary expertise domain. In contrast, humans who delegate predictions to a superior AI in a complementary expertise domain significantly increase appropriate reliance when the model systematically errs in the human expertise domain. Furthermore, we show that humans differentiate between error types and that this effect is conditional on the considered expertise domain. This is the first empirical exploration of choice independence and error types in the context of human-AI collaboration. Our results have broad and important implications for the future design, deployment, and appropriate application of AI systems.

@en

Crowd-powered conversational systems (CPCS) solicit the wisdom of crowds to quickly respond to on-demand users' needs. The very factors that make this a viable solution - -such as the availability of diverse crowd workers on-demand - - also lead to great challenges. The ever-changing pool of online workers powering conversations with individual users makes it particularly difficult to generate contextually consistent responses from a single user's standpoint. To tackle this, prior work has employed conversational facts extracted by workers to maintain a global memory, albeit with limited success. Through a controlled experiment, we explored if a conversational agent, dubbed ContextBot, can provide workers with the required context on the fly for successful completion of affective support tasks in CPCS, and explore the impact of ContextBot on the response quality of workers and their interaction experience. To this end, we recruited workers (N=351) from the Prolific crowd-sourcing platform and carried out a 3×3 factorial between-subjects study. Experimental conditions varied based on (i) whether or not context was elicited and informed by motivational interviewing techniques (MI-adherent guidance, general guidance, and no guidance), and (ii) different conversational entry points for workers to produce responses (early, middle, and late). Our findings show that: (a) workers who entered the conversation earliest were more likely to produce highly consistent responses after interacting with ContextBot; (b) showed better user experience after they interacted with ContextBot with a long chat history to surf; (c) produced more professional responses as endorsed by psychologists; (d) and that interacting with ContextBot through task completion did not negatively impact workers' cognitive load. Our findings shed light on the implications of building intelligent interfaces for scaffolding strategies to preserve consistency in dialogue in CPCS.

@en

Recent advances in the performance of machine learning algorithms have led to the adoption of AI models in decision making contexts across various domains such as healthcare, finance, and education.Different research communities have attempted to optimize and evaluate human-AI team performance through empirical studies by increasing transparency of AI systems, or providing explanations to aid human understanding of such systems.However, the variety in decision making tasks considered and their operationalization in prior empirical work, has led to an opacity around how findings from one task or domain carry forward to another.The lack of a standardized means of considering task attributes prevents straightforward comparisons across decision tasks, thereby limiting the generalizability of findings.We argue that the lens of ‘task complexity’ can be used to tackle this problem of under-specification and facilitate comparison across empirical research in this area.To retrospectively explore how different HCI communities have considered the influence of task complexity in designing experiments in the realm of human-AI decision making, we survey literature and provide an overview of empirical studies on this topic.We found a serious dearth in the consideration of task complexity across various studies in this realm of research.Inspired by Robert Wood’s seminal work on the construct, we operationalized task complexity with respect to three dimensions (component, coordinative, and dynamic) and quantified the complexity of decision tasks in existing work accordingly.We then summarized current trends and proposed research directions for the future.Our study highlights the need to account for task complexity as an important design choice.This is a first step to help the scientific community in drawing meaningful comparisons across empirical studies in human-AI decision making and to provide opportunities to generalize findings across diverse domains and experimental settings.

@en

Conversational User Interfaces (CUIs) have been argued to have advantages over traditional GUIs due to having a more human-like interaction. The growing popularity of conversational agents has enabled humans to interact with machines more naturally. There is an increasing familiarity among people with conversational interactions mediated by technology due to the widespread use of mobile devices and messaging services and a hungry market for conversational agents. Based on the recent advances in conversational AI, as a result of the proliferation of large language models, the signs are that the future of human-computer interaction will have a significant conversational component. Today, over two-thirds of the population on our planet has access to the Internet, with ever-lowering barriers to accessibility. This tutorial will showcase the benefits of employing novel conversational interfaces for crowd computing, human-AI decision making, health and well-being, and information retrieval. Given the widespread adoption of AI systems across several domains, we will discuss the potential of conversational interfaces in facilitating and mediating people's interactions with AI systems. The tutorial will include interactive elements and discussions and provide participants with insights to inform the design of effective conversational interfaces.

@en

Time constraints are commonplace in our daily lives. While literature in recent years from the Information Retrieval (IR) community has increased our understanding of the effects of time constraints on search, practical effects on search outcomes have rarely been evaluated. Little is known about how different search interfaces influence search outcomes and experiences in time-constrained search. This constitutes a knowledge gap that we aim to address in our work. Through a pre-registered 4x4 between-subjects crowdsourced user study, we investigate the influence of four different interfaces (list view, grid-based view, absence of result snippets, and linear scanning pattern view) on search outcomes and experiences under imposed time constraints (no constraint and constraints at two, five, and eight minutes). Results from our study indicate that user task performance is considerably affected by time constraints. In addition, as time constraints are tightened, a trade-off between querying rates and click depths arises. While no interaction effects between SERP interfaces and time constraints were ultimately found, findings from this study form an essential foundation for future work on how search result presentation may assist those searchers under strict time constraints.@en

AI systems are increasingly being used to support human decision making. It is important that AI advice is followed appropriately. However, according to existing literature, users typically under-rely or over-rely on AI systems, and this leads to sub-optimal team performance. In this context, we investigate the role of stated system accuracy by contrasting the lack of system information with the presence of system accuracy in a loan prediction task. We explore how the degree to which humans understand system accuracy influences their reliance on the AI system, by investigating numeracy levels and with the aid of analogies to explain system accuracy in a first of its kind between-subjects study (N=281). We found that explaining the stated accuracy of a system using analogies failed to help users rely on the AI systemappropriately (i.e., the tendency of users to rely on the system when the system is correct, or on themselves otherwise). To eliminate the impact of subjective attitudes towards analogy domains, we conducted a within-subjects study (N=248) where each participant worked on tasks with analogy-based explanations from different domains. Results from this second study confirmed that explaining stated accuracy of the system with analogies was not sufficient to facilitate appropriate reliance on the AI system in the context of loan prediction tasks, irrespective of individual user differences. Based on our findings from the two studies, we reason that the under-reliance on the AI system may be a result of users' overestimation of their own ability to solve the given task. Thus, although familiar analogies can be effective in improving the intelligibility of stated accuracy of the system, an improved understanding of system accuracy does not necessarily lead to improved system reliance and team performance.

@en

Good mental health is imperative for one’s wellbeing. While clinical mental disorder treatments exist, self-care is an essential aspect of mental health. This paper explores the use and perceived trust of conversational agents, chatbots, in the context of crowdsourced self-care through a between-subjects study (N = 80). One group used a standalone system with a conventional web interface to discover self-care methods. The other group used the same system wrapped in a chatbot interface, facilitating utterances and turn-taking between the user and a chatbot. We identify the security and integrity of the systems as critical factors that affect users’ trust. The chatbot interface scored lower on both these factors, and we contemplate the potential underlying reasons for this. We complement the quantitative data with qualitative analysis and synthesize our findings to identify suggestions for using chatbots in mental health contexts.

@en

AI systems are increasingly incorporated into human decision-making. Yet, human decision-makers are often affected by their cognitive biases. In critical settings, such as medical diagnosis, criminal judgment, or information consumption, these cognitive biases hinder optimal decision outcomes, thereby resulting in dangerous decisions and negative societal impact. The use of AI systems can amplify and exacerbate cognitive biases in their users. In this workshop, we seek to foster discussions on ongoing research around cognitive biases in human-AI collaboration and identify future research directions to understand, quantify, and mitigate the effects of cognitive biases. We will explore cognitive biases appearing in various contexts of human-AI collaboration: what can cause them?; how can we measure, model, mitigate, and manage cognitive biases?; and how can we utilise cognitive biases for the greater good? We will reflect on workshop discussions to form a research community around cognitive biases and bias-aware systems.

@en