The noticeability of behavioral changes of a conversational agent

An evaluation of an agent-based social skills training system

Bachelor thesis (2023)

Authors

A. Zaheer Electrical Engineering, Mathematics and Computer Science

Contributors

W.P. Brinkman Interactive Intelligence - EEMCS (mentor)
M. Al Owayyed Interactive Intelligence - EEMCS (mentor)
E. Eisemann Computer Graphics and Visualisation - EEMCS (graduation committee member)
Faculty
Electrical Engineering, Mathematics and Computer Science, Electrical Engineering, Mathematics and Computer Science
Copyright: © 2023 Ashma Zaheer
More Info
expand_more

Published Date

03-07-2023

Language

English

Reuse Rights

Other than for strictly personal use, it is not permitted to download, forward or distribute the text or part of it, without the consent of the author(s) and/or copyright holder(s), unless the work is under an open content license such as Creative Commons.

Faculty

Electrical Engineering, Mathematics and Computer Science

Abstract

Helpline counselors can be trained faster using agent-based social skills training systems. These systems utilize conversational agents that simulate interactions with users and provide feedback. This research evaluates the noticeability of behavior changes in such a conversational agent. By examining participants' ability to detect and interpret these changes, valuable insights can be gained to enhance the agent's representation of real-life scenarios and improve its effectiveness as a training tool.

A mixed-methods approach was employed, combining quantitative data using descriptive statistics and qualitative data through content analysis. The quantitative analysis revealed a precision score of 93% and a recall score of 36%, indicating a higher accuracy in correctly identifying behavior changes compared to capturing all instances of change. The qualitative analysis revealed that participants' perceptions fell within four distinct categories: communication style, positive emotion, negative emotion, and tone & attitude. Participants demonstrated heightened attentiveness to changes in positive behavior, particularly instances of happiness expressed by the agent, and displayed less attention towards changes in negative behavior.

In conclusion, this research demonstrates that behavior changes in the conversational agent are noticeable, albeit with varying degrees of attention across different types of behavior. Further exploration with a larger sample size is warranted to ascertain the extent of noticeability. The diverse range of behavior changes indicates the agent's capability to adapt and exhibit visible shifts in behavior. These insights can inform the refinement and development of agent-based training systems, ultimately enhancing the training experiences of chat-based helpline counselors.

Files

Final_Paper.pdf
(pdf | 0.966 Mb)