Designing with AI : Video to Mindmaps

Abstract

In today’s world, products are increasingly intertwined with our daily lives, becoming more intricate. To design these interactions effectively, designers need to comprehend both the people involved in the activities and the technical systems. As user-centered design spreads, the focus shifts from viewing design as problemsolving to recognizing it as the social creation of new possibilities. This shift highlights the importance of users and their everyday lives, as well as their interactions with the product in designing.
Videos serve as a powerful tool for learning about users, their behaviors, and interactions. Video Based Design (VBD) involves analyzing videos to interpret, infer, question actions, and understand the user’s thought process. However, this approach can extend the time and analytical rigor needed due to the extensive analysis required to draw valid conclusions from the video content. To streamline this process and enhance researchers’ workflow, we prototyped a tool that can produce mind maps from videos. These mind maps serve as a visual representation of the video’s key concepts in an organized manner. This tool utilizes Large Language Models (LLMs) to process multimodal video inputs and output the generated mind maps.
To assess the impact of mind maps on the video-based design process, an experimental study was conducted involving 28 designers. Participants were asked to watch two videos from different contexts and then engage in activities using mind maps generated by both the LLM and a human designer. Through quantitative and qualitative analysis of the study data, we gained valuable insights, identified strengths and weaknesses, and proposed design enhancements for the next version of the tool.