Collaborative search is "the retrieval and sensemaking of information by multiple users with the same information need". The experience of collaborative search can be improved by employing a chatbot which is "a program designed to counterfeit a smart communication". Chatbot makes it easier for the users to be aware of each other's activities and to share, communicate and reach consensus during collaborative search and makes it more enjoyable. However, the prior research has been limited by using a single-user web search engine and a dedicated messaging platform. It is unknown whether the observed benefits are maintained in a collaborative search system where several collaborative tools exist, such as shared query history, bookmark, and built-in messaging capability. Hence, here we aim to explore how chatbot affects users' behavior on a collaborative search system and users' perceptions of the chatbot. We implemented ChatX, a chatbot agent that monitors the group chat and guides users through the search tasks. To evaluate it, we conducted a user study with X users in the domain of restaurant recommendations and local attraction searching. We found that ChatX does not significantly reduce users' search effort, but it improves the overall collaborative search experience and gives the user a highly favorable impression.