In the shift towards human-centered manufacturing, our two-year longitudinal study investigates the real-world impact of deploying Cognitive Assistants (CAs) in factories. The CAs were designed to facilitate knowledge sharing among factory operators. Our investigation focused on smartphone-based voice assistants and LLM-powered chatbots, examining their usability and utility in a real-world factory setting. Based on the qualitative feedback we collected during the deployments of CAs at the factories, we conducted a thematic analysis to investigate the perceptions, challenges, and overall impact on workflow and knowledge sharing.
Our results indicate that while CAs have the potential to significantly improve efficiency through knowledge sharing and quicker resolution of production issues, they also introduce concerns around workplace surveillance, the types of knowledge that can be shared, and shortcomings compared to human-to-human knowledge sharing. Additionally, our findings stress the importance of addressing privacy, knowledge contribution burdens, and tensions between factory operators and their managers.@en

This project is an exploration of potential applications of Conversational AI Agents (CAIA) for industrial maintenance. Specifically, it involves the scientific validation and development of a CAIA for a promising application: the automatic creation of information-rich maintenance reports by conversing with a technician while they perform industrial maintenance. The choice for this application was based on literature research, (in-situ) context analyses and a review of CAIA frameworks and design guidelines. These revealed that maintenance workers rely heavily on their own experience and intuition when solving problems but mechanisms for capturing and accessing this were non-existent. This knowledge is highly valuable and can represent a significant part of a company’s worth. Furthermore, maintenance technicians reported that the existing reporting mechanisms were a nuisance. Lastly, audits revealed that maintenance reports were frequently incomplete or of poor quality. In turn, CAIAs (Conversational AI Agents), have various affordances that make them well-suited to the context of industrial maintenance. They are (1) hands and gaze-free, (2) highly efficient (faster than writing or typing, facilitate multitasking and they provide faster access to specific information), (3) they can adapt to the skill level of the user and (4) impose a minimal cognitive load.
A between-subjects experiment with 24 participants, which involved changing a bicycle inner tube, was used to test three hypotheses regarding the potential value of the application. All three hypotheses compare using a CAIA for reporting whilst performing maintenance, as opposed to writing the reports on paper afterwards. They posited that using the CAIA would result in (1) reports of higher quality (more information relevant to the understanding of the task), (2) time saving, and (3) a lower perceived workload (NASA TLX). T-tests confirmed that all three hypotheses were true. These results indicate that using a CAIA for live-reporting has a clear value proposition for the industrial maintenance domain. Critically, it demonstrated that it could facilitate the capture of valuable “expert knowledge”. Future research could explore integrating multi-modal information capture (e.g. through smart-glasses), identify additional uses for the captured data (e.g. for prescriptive maintenance or providing tips to maintenance technicians) and improve the functionality and usability of the existing application. A prototype was built using the open-source frameworks, Rasa and Mycroft, to demonstrate the technical feasibility of the functional requirements. These requirements include (1) uttering “continuers” in response to the user describing their work, (2) tracking conversational context, (3) asking for clarifications when there is a lack of mutual understanding and (4) asking for status updates when the user is silent for more than a set amount of time. The main challenges for the future development of the prototype are (1) reducing the response time of the CAIA, (2) the accuracy of the intent classifier and entity extractor and (3) improving the handling of fragmented/lengthy user input. Some of the prototype’s features rely on inflexible, hard-coded logic, therefore, future work should collect more conversational data and explore the use of machine learning algorithms.