Endowing machines with social competence is not only a science fiction theme. It is also a long-held goal in computer science. Machines have changed how we work, communicate, and do art, science, and engineering, but they have had little effect on one of our core human needs: social interaction. Although digital communication has changed the way we interact with others, machines have arguably done little to enhance the quality of our face-to-face interactions and are seldom seen as tools to help us improve the way we interact with others. This is in part due to their lack of social competence thus far.

A crucial stepping stone towards social competence and the ability to display empathy is the ability to assess social experience. Social experience refers to internal states reflecting an individual's perception of a social situation, like enjoying a conversation or feeling attracted to someone they are interacting with. Social experience variables are hard to study because they are not directly observable and change over time. Researchers must rely on self-reports or third-party assessments (annotations). Algorithms for assessment of social experience generally take one of two approaches: 1) Direct modeling of the relationship between raw/derived signals and experience variables, utilizing sensor readings or outputs of detectors and feature extractors; and 2) intermediate modeling/detection of discrete actions performed during interactions (ie. speaking, laughter, gesturing).

In this thesis dissertation, we focus on in-the-wild mingling setting, where subjects are standing and are free to form and switch conversation groups as they desire. Data collection and annotation are paid special attention due to their relevance in a nascent field and the nuance involved in collecting and annotating social signals. Because the goal is to study machine social perception in real-life settings, interactions are not scripted and instrumentation is kept to a minimum.

We start with work concerned with the direct assessment of social experience, in this case of attraction, by exploring the predictive power of body acceleration. By analyzing accelerometer data from speed dating interactions, we investigate how the intensity and variations in body movement relate to self-reported attraction levels. This study sheds light on the predictive power of synchrony, mimicry, and convergence estimates for predicting attraction, and potentially other constructs related to affiliation.

We then address the detection of speaking, an action of wide interest in social signal processing due to the relevance of turn-taking in social experience. We address the limitations posed by visual cross-contamination in crowded mingling settings. We introduce a model that employs accelerometer readings and body poses to enhance the robustness of speaking status detection in a complex scene, with multiple interactions occurring simultaneously.

The dissertation also presents two novel datasets: ConfLab and REWIND, each serving a unique purpose. ConfLab, collected during a conference, is notable for its annotations of body joints, and improvements to the sensor setup resulting in increased data fidelity. Such methodological contributions to enable efficient and high-quality data collection are increasingly valuable given the scarcity of social interaction datasets, particularly in mingling settings. REWIND, gathered at a business networking event, stands out with its high-quality individual audio recordings, useful for the cross-modal study of multimodal signals such as speaking or laughter.

In a similar line, we present the Covfee software framework. Covfee challenges existing annotation methodologies by introducing and studying interfaces for continuous annotation for keypoints and actions. This framework was instrumental in efficiently processing the vast amounts of data collected in studies like ConfLab by streamlining the annotation process.

Also building on the Covfee framework, the dissertation culminates in an exploration of laughter annotation across different modalities. By comparing laughter annotations acquired in different conditions, the research highlights the complexities and nuances involved in interpreting social signals across different sensory inputs. We challenge the assumption that laughter intensity should be considered a property of the laughter episode. Instead, we find evidence that laughter evaluations differ significantly depending on the modalities available to the observer and that modalities with higher agreement will not necessarily result in the highest model performance. These results not only contribute to the study of laughter detection but also provide valuable insights for future research on multimodal social signal processing.

In summary, this dissertation weaves together a series of methodological contributions and novel findings, often derived from these new methods, each contributing to further our understanding of how to best train machines for social understanding and competence.@en

This work gives a method which can use both motion and shape information for drinking action detection. Drinking action is differ from the other gestures. A complete drinking action duration could be divided into 3 different stages. Motion-based features and shape-based features are complement with each other in each stage. By using this feature, we have defined a fusion strategy that can use both the advantage of the strength of each part at a different stage.

Person re-identification (re-ID) is a task that aims to associate the same people across different cameras. One of the many important problems a person re-ID system has to address in order to achieve good performance is the feature misalignment problem. Past research has attempted to address this problem by using attention networks, pose-estimation modules, or semantic segmentation networks. However, they all eventually tend to pool these features to a single feature embedding, thereby not distinguishing regions with different semantic meanings such as the head, torso, and lower body. Most approaches also do not make use of all the information available throughout multiple layers (stages) of the feature extractor. Furthermore, although these additional features are used to provide extra information to the re-ID network during training, they do not take into account the importance of different regions of the image due to, for example, occlusion. To circumvent these problems, we propose a network that is capable of extracting regional feature embeddings that are associated with specific body parts of an identity, i.e., head, upper-body, lower-body, shoes, and foreground image. We extract these features from multiple stages of a feature extractor using a semantic-segmentation module. We then use multi-branch learning to ensure that these features are independently optimized by introducing separate modules (branches) for each regional feature embedding. To increase the robustness of the model, we also propose a novel testing strategy that makes use of the importance and visibility of specific body parts in both the query and gallery images in order to calculate a ranking list. Finally, to address the current lack of datasets that contain images from overhead face-down cameras, we introduce a new dataset named MatchNMingle-reID. Because of the viewpoint of the cameras, this dataset presents unique challenges that are not seen in current datasets and opens possibilities to create more generalizable models that can effectively address the feature misalignment problem.