Modern factories showlittle resemblance to the assembly lines froma century ago. Nowadays, a single human might be responsible for setting up, operating, and maintaining a complex chain of machines. This shift has made the work of factory operators more cognitively demanding, requiring constant monitoring and problem-solving. Key to tackling these challenges is the effective collaboration of human operators in exchanging ideas and knowledge, from high-level problem-solving strategies to solutions for emerging issues. Yet, the heightened productivity requirements and small shifts mean fewer opportunities to share this knowledge face-to-face and reporting practices are deprioritized. Thus, the manufacturing industry faces a knowledge management crisis.
This dissertation investigates the integration of conversational AI assistants into manufacturing settings to facilitate knowledge sharing among factory operators. Capitalizing on recent advancements in Natural Language Processing (NLP), particularly in Large Language Models (LLMs), this research investigates the designing and evaluation of conversational AI tools that efficiently capture and share human knowledge on the factory floor while addressing operator needs and concerns. The introduction of conversational AI assistants for knowledge sharing—referred to as cognitive assistants (CA) in this work—in factory environments promises significant benefits but comes with numerous challenges....@en

Large Language Models (LLMs) are expected to significantly impact various socio-technical systems, offering transformative possibilities for improved interaction between humans and technology. However, their integration poses complex challenges due to the intricate interplay between societal structures, human behaviour, and technological innovation. This research explores these multifaceted challenges, emphasising the need for a human-centered approach in integrating LLMs to ensure that technological advancements are aligned with ethical standards and societal needs. Utilizing a structured methodology comprising a workshop, literature analysis, and expert collaborations, the study uses a multi-dimensional human-centered AI framework to guide the responsible integration of LLMs. Key insights include the importance of inclusive data, considering unintended consequences, maintaining privacy, and respecting intellectual property rights. The paper identifies and advocates for principles like human-in-the-loop, continuous longitudinal studies, proactive awareness campaigns, and regular audits to develop LLMs that are ethically sound, adaptable, and effectively integrated into various socio-technical systems, thus addressing user needs and broader societal impacts. The paper also underlines the importance of collaboration among academia, industry, and policymakers to develop LLMs that are ethically aligned, socially beneficial, and adaptable to future societal needs. The findings offer valuable insights into the strategic integration of LLMs, advocating for a broader research perspective beyond industrial motivations to fully understand and leverage LLMs in socio-technical landscapes.

@en

Recent advances in natural language processing enable more intelligent ways to support knowledge sharing in factories. In manufacturing, operating production lines has become increasingly knowledge-intensive, putting strain on a factory's capacity to train and support new operators. This paper introduces a Large Language Model (LLM)-based system designed to retrieve information from the extensive knowledge contained in factory documentation and knowledge shared by expert operators. The system aims to efficiently answer queries from operators and facilitate the sharing of new knowledge. We conducted a user study at a factory to assess its potential impact and adoption, eliciting several perceived benefits, namely, enabling quicker information retrieval and more efficient resolution of issues. However, the study also highlighted a preference for learning from a human expert when such an option is available. Furthermore, we benchmarked several commercial and open-sourced LLMs for this system. The current state-of-the-art model, GPT-4, consistently outperformed its counterparts, with open-source models trailing closely, presenting an attractive option given their data privacy and customization benefits. In summary, this work offers preliminary insights and a system design for factories considering using LLM tools for knowledge management.@en

Learning to operate a complex system, such as an agile production line, can be a daunting task. The high variability in products and frequent reconfigurations make it difficult to keep documentation up-to-date and share new knowledge amongst factory workers. We introduce CLAICA, a Continuously Learning AI Cognitive Assistant that supports workers in the aforementioned scenario. CLAICA learns from (experienced) workers, formalizes new knowledge, stores it in a knowledge base, along with contextual information, and shares it when relevant. We conducted a user study with 83 participants who performed eight knowledge exchange tasks with CLAICA, completed a survey, and provided qualitative feedback. Our results provide a deeper understanding of how prior training, context expertise, and interaction modality affect the user experience of cognitive assistants. We draw on our results to elicit design and evaluation guidelines for cognitive assistants that support knowledge exchange in fast-paced and demanding environments, such as an agile production line.

@en

Many industries face the challenge of capturing workers' knowledge to share it, particularly tacit knowledge. The operation of complex systems such as a manufacturing line is knowledge-intensive. Considering this knowledge's breadth and dynamic nature, existing knowledge-sharing solutions are inefficient and resource intensive. Conversational user interfaces are an efficient way to convey information that mimics how humans share knowledge; however, we know little about how to design them specifically for knowledge sharing, especially regarding tacit knowledge. In this work, we present an intelligent assistant that we have developed to support the elicitation of tacit knowledge from workers through systematic reflection. The system can interact with workers by voice or text and generate visualizations of shop floor data to support reflective prompts.

@en

As agile manufacturing expands and workforce mobility increases, the importance of efficient knowledge transfer among factory workers grows. Cognitive Assistants (CAs) with Large Language Models (LLMs), like GPT-3.5, can bridge knowledge gaps and improve worker performance in manufacturing settings. This study investigates the opportunities, risks, and user acceptance of LLM-powered CAs in two factory contexts: textile and detergent production. Several opportunities and risks are identified through a literature review, proof-of-concept implementation, and focus group sessions. Factory representatives raise concerns regarding data security, privacy, and the reliability of LLMs in high-stake environments. By following design guidelines regarding persistent memory, real-time data integration, security, privacy, and ethical concerns, LLM-powered CAs can become valuable assets in manufacturing settings and other industries.

@en

Operating a complex and dynamic system, such as an agile manufacturing line, is a knowledge-intensive task. It imposes a steep learning curve on novice operators and prompts experienced operators to continuously discover new knowledge, share it, and retain it. In practice, training novices is resource-intensive, and the knowledge discovered by experts is not shared effectively. To tackle these challenges, we developed an AI-powered pervasive system that provides cognitive augmentation to users of complex systems. We present an AI cognitive assistant that provides on-the-job training to novices while acquiring and sharing (tacit) knowledge from experts. Cognitive support is provided as dialectic recommendations for standard work instructions, decision-making, training material, and knowledge acquisition. These recommendations are adjusted to the user and context to minimize interruption and maximize relevance. In this article, we describe how we implemented the cognitive assistant, how it interacts with users, its usage scenarios, and the challenges and opportunities.@en

Emoji have become an essential part of modern communication, helping to convey emotions and tone quickly and concisely. Emoji used by humans and Intelligent Agents (IA) have been shown to affect people’s decision making intentions, suggesting they could be used to manipulate users to follow their advice. We present a mixed-methods crowdsourcing study (N = 194) that shows that adherence to an IA’s recommendation and user experience are not affected by emoji when used in a positive, collaborative way. However, we demonstrate that explanations provided by an IA do increase adherence to its recommendation.

@en

Many industries are facing the challenge of how to capture workers' knowledge such that it can be shared, in particular tacit knowledge. The operation of complex systems such as a manufacturing line is knowledge-intensive, especially if the operator must frequently reconfigure it for different products. Considering the breadth and dynamic nature of this knowledge, existing solutions for sharing knowledge (e.g., word-of-mouth, issue reports, document creation, and decision support systems) are inefficient and/or resource-intensive. Conversational user interfaces are an efficient way to convey information that mimics the way humans share knowledge; however, we know little about how to design them specifically for this purpose, especially regarding tacit knowledge. In this work, my main goal is to investigate how a cognitive assistant can be designed to facilitate (tacit) knowledge transfer between users of dynamic complex systems. I aim to achieve this by outlining the design requirements, challenges, and opportunities in factories; by collaboratively designing, implementing, and evaluating a cognitive assistant for sharing knowledge; studying the effects of design characteristics on aspects such as user experience; and finally, creating a set of design guidelines.

@en

Maintaining a complex system, such as a modern production line, is a knowledge-intensive task. Many firms use maintenance reports as a decision support tool. However, reports are often poor quality and tedious to compile. A Conversational User Interface (CUI) could streamline the reporting process by validating the user's input, eliciting more valuable information, and reducing the time needed. In this paper, we use a Technology Probe to explore the potential of a CUI to create instructional maintenance reports. We conducted a between-groups study (N = 24) in which participants had to replace the inner tube of a bicycle tire. One group documented the procedure using a CUI while replacing the inner tube, whereas the other group compiled a paper report afterward. The CUI was enacted by a researcher according to a set of rules. Our results indicate that using a CUI for maintenance reports saves a significant amount of time, is no more cognitively demanding than writing a report, and results in maintenance reports of higher quality.

@en

Conversational agents (CA) occasionally fail to understand the user's intention or respond inappropriately due to natural language complexity. These conversational breakdowns can happen because of low intent and entity prediction confidence scores. A promising repair strategy in such cases is that the CA proposes to users likely alternatives to proceed. If one of these options matches the user's intention, the breakdown is repaired successfully. We propose that successful repairs should be followed by a learning mechanism to minimize future breakdowns. After a successful repair, the CA, user, or both can learn each other's specific phrasing. This prevents similar phrasings from causing reoccurring breakdowns. We compared user preferences for these learning mechanisms in a scenario-based study with manufacturing workers (). Our result showed that users first prefer to share the learning burden with the CA (61.3%), followed by entirely outsourcing the learning burden to the CA (60.7%) as opposed to themselves.

@en