In the shift towards human-centered manufacturing, our two-year longitudinal study investigates the real-world impact of deploying Cognitive Assistants (CAs) in factories. The CAs were designed to facilitate knowledge sharing among factory operators. Our investigation focused on smartphone-based voice assistants and LLM-powered chatbots, examining their usability and utility in a real-world factory setting. Based on the qualitative feedback we collected during the deployments of CAs at the factories, we conducted a thematic analysis to investigate the perceptions, challenges, and overall impact on workflow and knowledge sharing.
Our results indicate that while CAs have the potential to significantly improve efficiency through knowledge sharing and quicker resolution of production issues, they also introduce concerns around workplace surveillance, the types of knowledge that can be shared, and shortcomings compared to human-to-human knowledge sharing. Additionally, our findings stress the importance of addressing privacy, knowledge contribution burdens, and tensions between factory operators and their managers.@en

Large Language Models (LLMs) are expected to significantly impact various socio-technical systems, offering transformative possibilities for improved interaction between humans and technology. However, their integration poses complex challenges due to the intricate interplay between societal structures, human behavior, and technological innovation. This research explores these multifaceted challenges, emphasizing the need for a human-centered approach in integrating LLMs to ensure that technological advancements are aligned with ethical standards and societal needs. Utilizing a structured methodology comprising a workshop, literature analysis, and expert collaborations, the study uses a multi-dimensional human-centered AI framework to guide the responsible integration of LLMs. Key insights include the importance of inclusive data, considering unintended consequences, maintaining privacy, and respecting intellectual property rights. The paper identifies and advocates for principles like human-in-the-loop, continuous longitudinal studies, proactive awareness campaigns, and regular audits to develop LLMs that are ethically sound, adaptable, and effectively integrated into various socio-technical systems, thus addressing user needs and broader societal impacts. The paper also underlines the importance of collaboration among academia, industry, and policymakers to develop LLMs that are ethically aligned, socially beneficial, and adaptable to future societal needs. The findings offer valuable insights into the strategic integration of LLMs, advocating for a broader research perspective beyond industrial motivations to fully understand and leverage LLMs in socio-technical landscapes.@en

This project is an exploration of potential applications of Conversational AI Agents (CAIA) for industrial maintenance. Specifically, it involves the scientific validation and development of a CAIA for a promising application: the automatic creation of information-rich maintenance reports by conversing with a technician while they perform industrial maintenance. The choice for this application was based on literature research, (in-situ) context analyses and a review of CAIA frameworks and design guidelines. These revealed that maintenance workers rely heavily on their own experience and intuition when solving problems but mechanisms for capturing and accessing this were non-existent. This knowledge is highly valuable and can represent a significant part of a company’s worth. Furthermore, maintenance technicians reported that the existing reporting mechanisms were a nuisance. Lastly, audits revealed that maintenance reports were frequently incomplete or of poor quality. In turn, CAIAs (Conversational AI Agents), have various affordances that make them well-suited to the context of industrial maintenance. They are (1) hands and gaze-free, (2) highly efficient (faster than writing or typing, facilitate multitasking and they provide faster access to specific information), (3) they can adapt to the skill level of the user and (4) impose a minimal cognitive load.
A between-subjects experiment with 24 participants, which involved changing a bicycle inner tube, was used to test three hypotheses regarding the potential value of the application. All three hypotheses compare using a CAIA for reporting whilst performing maintenance, as opposed to writing the reports on paper afterwards. They posited that using the CAIA would result in (1) reports of higher quality (more information relevant to the understanding of the task), (2) time saving, and (3) a lower perceived workload (NASA TLX). T-tests confirmed that all three hypotheses were true. These results indicate that using a CAIA for live-reporting has a clear value proposition for the industrial maintenance domain. Critically, it demonstrated that it could facilitate the capture of valuable “expert knowledge”. Future research could explore integrating multi-modal information capture (e.g. through smart-glasses), identify additional uses for the captured data (e.g. for prescriptive maintenance or providing tips to maintenance technicians) and improve the functionality and usability of the existing application. A prototype was built using the open-source frameworks, Rasa and Mycroft, to demonstrate the technical feasibility of the functional requirements. These requirements include (1) uttering “continuers” in response to the user describing their work, (2) tracking conversational context, (3) asking for clarifications when there is a lack of mutual understanding and (4) asking for status updates when the user is silent for more than a set amount of time. The main challenges for the future development of the prototype are (1) reducing the response time of the CAIA, (2) the accuracy of the intent classifier and entity extractor and (3) improving the handling of fragmented/lengthy user input. Some of the prototype’s features rely on inflexible, hard-coded logic, therefore, future work should collect more conversational data and explore the use of machine learning algorithms.