This study explores the influence of interrater agreement measures and affect representation schemes in automatic affect prediction systems using physiological signals. These systems often use supervised learning and require unambiguous and objective labeling, a challenge when multiple human annotators are involved, which can affect model performance.
The research involved the first part of a two-stage process: systematically reviewing datasets and their characteristics concerning interrater agreement on the affective interpretation of physiological signals. This stage established a reliable foundation for the second step: a future analysis of model performance reported in technical papers utilizing these datasets. The main takeaways were that the number of raters varies significantly over datasets and the complexity introduced by combining affect representation schemes can negatively affect interrater agreement.

Recognizing facial emotions is key for social interaction, yet the subjective nature of emotion labeling poses challenges for automatic facial affect prediction. Variability in how individuals interpret emotions leads to uncertainty in training data for machine learning models. While multiple raters and interrater agreement (IRA) measures are used to address this, the extent of their use and their impact on dataset reliability is not well understood. This systematic literature review investigates the methodologies used to measure IRA in facial affect recognition datasets. Concrete eligibility and feasibility criteria were applied, and it resulted in 47 papers being retrieved from Scopus, Web of Science, IEEExplore, and ACM Digital Library. Data on affect states, affect representation schemes (ARS), and IRA methodologies used by the datasets and their corresponding papers were extracted to provide a comprehensive overview and allow a detailed analysis. Clear correlation was not found in between ARS and IRA, but the retrieved data showed that Fleiss' kappa was the most popular methodology over time but also in the recent years.

Emotional datasets for automatic affect prediction usually employ raters to annotate emotions or verify the annotations. To ensure the reliability of these raters some use interrater agreement measures, to verify the degree to which annotators agree with each other on what they rate. This systematic review explores what kind of interrater agreement measures are used in emotional speech corpora. The affective states, the affect representation schemes, and the collection method of the datasets as well as the popularity of these measures were investigated. Scopus, IEEE Xplore, Web of Science, and ACM digital library were used to extract papers that describe the creation of datasets. 45 papers were included in the review. The review concludes that the interrater agreement measures used, are highly dependent on the collection method for speech and the affect representation schemes. It was found that there is no standardized way to measure interrater agreement. Datasets that use actors to record emulated emotions mostly use recognition rate as their interrater agreement measures. Datasets that use a dimensional representation scheme often compute the mean agreement of the raters and the standard deviation of that measure to check the interrater agreement. Datasets that do not use actors nor are dimensional use a plethora of different measures such as probabilistic computing of agreement, or majority agreement measures, but a large amount use no measures at all.

With the rise in the number of human-computer interactions, the need for systems that can accurately infer and respond to users' emotions becomes increasingly important. One can achieve this by examining audio-visual signals, aiming to identify the underlying emotions from an individual's gestures, auditory cues, and surroundings. Such automatic affect prediction systems depend heavily on labeled datasets. However, the subjective nature of emotion interpretation often introduces uncertainty, making it challenging to create reliable and high-quality datasets. To combat this issue, researchers have tried employing multiple raters to judge the affective state of a person, while employing interrater agreement measures to monitor uncertainty. To this moment, it is still unclear to what extend does reaching a good level of interrater agreement impact the performance of audio-visual Automatic Affect Prediction models.
As a first step towards understanding the potential influences on performance, this paper conducts a systematic literature review to investigate the objective annotation procedure in audio-visual databases. The survey extracted relevant literature from 4 scientific databases: Scopus, IEEE Xplore, Web of Science and ACM Digital Library. The results are aggregated from 55 papers and presented by following the PRISMA guidelines. They indicate that most databases use multiple annotators, a little more than half measure interrater agreement, and most train the raters to increase the uniformity of the labels.

Human-computer interaction has long been the focus of technological evolution; however, in order for this type of system to reach its peak potential, machines must recognize that humans are constantly influenced by emotions. Text affective content analysis models are one attempt to integrate human psychology into computers, trying to detect the emotion transmitted by written input. There are numerous approaches to implementing such systems, with supervised learning still popular. The challenge of creating textual affective datasets is not in the availability of records, as humanity has reached a peak in data production, especially text, but in ensuring the consistency of the annotations provided by humans when included in the process. This study conducts a systematic literature review focused on providing details of published corpora. The annotation process, as well as any trends in how it evolved, will be examined to obtain dataset particularities. The ultimate intent is to lay the groundwork for an ample study aimed at analyzing the relationship between interrater agreement levels and performance scores of models trained on these datasets. The relevant literature was extracted from 3 search engines: Scopus, IEEE Xplore, and Web of Science, with a focus on manually labeled written records that are not part of multimodal systems, resulting in an analysis of 41 datasets. According to the aggregations, when humans are recruited to perform this task, researchers are more likely to use multiple annotators and calculate the degree of agreement between them to ensure the data's reliability before using it. The conducting researchers are inclined to either train these people before the procedure or tailor the set of labels to potentially increase the uniformity of ratings. As a result, this paper highlights the variety of annotation process characteristics and points towards standardizing this task.

Understanding how users retrospectively evaluate their interactions with adaptive intelligent systems is crucial to improving their behaviours during interactions. Prior work has shown the potential to predict retrospective evaluations based on different real-time aspects of conversations, such as verbal cues and non-verbal behaviours. However, the relationship between how one retrospectively evaluates and the real-time evaluations in the moment of conversations remains unclear. This study investigates the relationship between real-time evaluations of a situation, using the Situational Interdependence Scale (SIS) framework, and its retrospective evaluations. We investigate the presence of the peak-end rule and a complex relationship that could be modelled using Long Short-Term Memory (LSTM) for each SIS dimension using the PACO dataset. Due to the absence of ground truth for real-time SIS evaluations, we also present a methodologically sound technical approach to utilize a Large Language Model (LLM) to estimate values for each SIS dimension for each spoken utterance in conversations. Analysis of the experiments revealed the absence of both the peak-end rule and an LSTM-modelled relationship across all dimensions of SIS. However, both types of models at least predict better than the average of the estimated real-time evaluation. This may be largely due to the inaccuracy of the estimated real-time SIS evaluations and the limited LLM’s capability of labelling real-time SIS in conversational data. Future works may focus on improving the annotation of real-time SIS evaluations through human annotation or human-supervised few-shot learning of LLM, using other modalities in combinations with verbal content, and exploring other predictive models.

Cognitive Impairment, commonly termed as Dementia, affects a large number of older adults. People with dementia (PwD) experience cognitive decline that impacts their ability to perform daily activities and maintain social connections. The number of PwD is expected to rise, and unfortunately, there is still no cure. A number of non drug therapies such as music and art therapy are used to stimulate neurons and slow down the progress of the disease. When symptoms become severe, full-time care may be needed, requiring a move to a nursing home for formal care. However, studies show that interaction with peers is limited in a care home, leading to loneliness and reduced quality of life. Caregivers organize group activities like singing and baking to alleviate these feelings, but their busy schedule can limit the frequency of these exercises. This begs the question: Can a robot support caregivers in the provision of group activities in a care home? This would allow caregivers more time to attend to the personal needs of the residents. It not only has the potential to reduce workload of caregivers but also creates opportunities for residents to interact with each other.

In this thesis, we propose the design of a robot which moderates a sing-along activity for a group of people living in a nursing home. An activity in the realm of music therapy is chosen since it is one of the non-drug therapies which has shown to have numerous benefits like improving memory recall, eliciting emotions and aiding relaxation. The robot session is curated to give opportunities for interaction with group members and with the robot too. While the song is playing and participants are singing along with the music, the robot performs engagement detection, and encourages those who do not seem involved.

Multiple ways are proposed to detect engagement in HRI, such as emotion recognition and estimation of head pose. These approaches have limitations in the scope of this thesis, particularly because these methods are not designed for groups of older adults residing in a care home. Given these limitations and our focus on the target group, we propose a novel engagement detection methodology, which leverages the presence of a robot and assigns an active role to it. Instead of relying on conventional passive engagement detection methods, this approach uses an interactive probing technique. The robot prompts users to perform certain actions, and whether or not they respond to it is monitored using pose estimation. This is termed robot engagement. This approach is combined with activity engagement, which indicates whether the participant is singing or not. With this hybrid technique, we aim to increase the efficacy of engagement detection. If a participant is detected as disengaged, either in the activity or in the robot, we provide personalized encouragement and motivation by addressing participants by name. Along with this, the robot also compliments those who actively participate, motivating them further...

Studies in Music Affect Content Analysis use varying emotion schemes to represent the states induced when listening to music. However, there are limited studies that explore the translation between these representation schemes. This paper explores the feasibility of using machine learning models to translate from a dimensional scheme of Valence, Energy and Tension, to a categorical emotion of Anger, Fear, Happy, Sad, or Tender, specifically in the context of musical stimuli. Additionally, this paper considers how the close proximity of certain emotions in the dimensional space, such as Fear and Anger, negatively influence the performance of translation models. This paper reflects on past studies and presents new results concluding feasible translations of music affect content, moreover, providing suggestions for future analysis.

Images possess the ability to convey a wide range of emotions, and extracting affective information from images is crucial for affect prediction systems. This process can be achieved through the application of machine learning algorithms. Categorical Emotion States (CES) and Dimensional Emotion Space (DES) are two typical models used for representing emotions. Moreover, the development of a mapping schema between these representations can contribute to benefit the research in AI and psychology. Consequently, this study focuses on investigating the feasibility of translating emotions from DES to CES. To accomplish this goal, relevant databases are identified and combined as the training data, and machine learning models, namely Naive Bayes, K-nearest Neighbors, and Decision Tree are employed to perform the classification task. The results indicate the superior performance of the K-nearest Neighbors classifier, exhibiting higher mean accuracy (60%) and low standard deviation (0.004) among all implemented classifiers. Overall, the translation from DES to CES offers several advantages, including a simplified and interpretable representation of emotions, as well as the provision of a common language for discussing and expressing emotions.

In human-human interactions, the majority of information is conveyed through body language, specifically facial expressions. Consequently, researchers have been interested in improving human-computer interactions through developing systems with automatic understanding of body language and facial expressions.  This technology is especially useful due to its broad range of applications in fields such as healthcare, education, and safety & security. Vision-based automatic affect recognition (AAR) systems aim to predict a subject’s affective state based on visual input such as image or video. These systems analyze and classify subjects’ facial expressions and body language using affect representation schemes (ARS), most often classified as either categorical or dimensional. This paper explores the current state of ARS used in vision-based AAR through a systematic literature review following PRISMA guidelines. We selected 53 papers from WebOfScience according to our eligibility criteria which included computer science papers written in English proposing a vision-based AAR system targeting single subjects, and excluded studies dealing exclusively with micro-expressions or group affect recognition. Additionally, given the time limitation imposed on this research we excluded papers that were not readily accessible with our TU Delft license, used multimodal input, or did not use a dataset included in our  predefined list. For this exploration we specifically look at the schemes used, the popularity and trends of usage, motivations, and psychological basis. From the 53 reviewed papers, all of the papers target utilitarian emotions using at least one discrete ARS. The most commonly used schemes classify affective states into happiness, sadness, fear, anger, surprise, and disgust. While the majority of papers are lacking in providing explicit reasoning for their choices, most ARS are based grounded in psychological theories. Our results show an established norm within this area of research. However, they also evidence a lack of displayed critical thought in the selection of schemes. This oversight limits potential for future AAR research.

Physiological signals, such as Electroencephalogram (EEG), Glavic Skin Response (GSR), or Body Temperature, are common inputs for Automatic Affect Recognition (AAR) systems. One of the crucial elements of AAR is the Affect Representation Scheme (ARS) used to define the affective states recognized by the system (e.g., happiness, sadness, fear, anger). Throughout the years many AAR reviews have been published. However, most of them do not include a detailed analysis of ARSs and the motivation behind them. This paper aims to fill this knowledge gap by performing a Systematic Literature Review (SLR) of Computer Science papers that propose a Physiological-signal-based AAR (PAAR) system. We explore how researchers discuss and choose an ARS and whether they base it on actual psychological theories. Eligible papers are retrieved from 4 databases: Web Of Science, IEEExplore, Scopus, and ACM Digital Library. Due to time limitations, the review is done rapidly and some additional search constraints are applied for feasibility. The most significant restrictions are: considering papers published since 2020 and performing experiments on specific benchmarking datasets. We take these constraints and their possible impact into consideration when interpreting the results. The presented review procedure can be stripped from the additional filters and reused for a full review. In total 115 papers are processed. The majority of papers introduce an EEG-based emotion recognition system. The analysis of the extracted information reveals that dimensional ARSs, in particular, Valence/Arousal model is the most popular. Moreover, authors often choose to reduce the dimensions to high/low values. Categorical ARSs are less frequent and usually are adopted from the dataset. Lastly, the authors do not provide extensive motivation for the choice of ARS and rarely refer to psychological theories.

Affective Video Content Analysis aims to automatically analyze the intensity and type of affect (emotion or feeling) that are contained in a video and are expected to arise in users while watching that video. This study aims to provide a systematic overview of various affect representation schemes utilised by researchers in the field of Affective Video Content Analysis and the reasons behind their choice. The main objectives of the study were to investigate the diversity of affect representation scheme types, their popularity over time, the basis of their selection, and the relationship between input data sources, in terms of direct and implicit analysis, and scheme types. Following the PRISMA guidelines to conduct a systematic literature review, a total of 45 papers were included in the study which were original journals and conference proceedings in the field of Affective Video Content Analysis and that were related only to Affective Movie Content Analysis published in English after 2008. Papers concerning Video Emotion Recognition were excluded from the review. The findings reveal that dimensional, categorical, and combined approaches are commonly used in this field, with the dimensional approach based on valence and arousal being the most prevalent. However, there is no significant trend in the popularity of affect representation scheme types over time. The study highlights the lack of clear motivation and explicit justifications for scheme selection, emphasizing the need for transparency and the inclusion of psychological theories as a basis for scheme choices. Additionally, the study found that audiovisual data was the most commonly used input compared to physiological signals and visible behavioural data. The study acknowledges its limitations, including time constraints and single researcher involvement, and suggests allocating more time and involving multidisciplinary teams for comprehensive insights.

There is a correlation between music and affect which researchers try to define and use in technology to improve healthcare and users' experience in music-related technology. However, since affect is a complex term there is not a specified way on how to represent different affective states in systems. A systematic literature review was performed to give an overview of affect representation schemes (ARS) used in music affect prediction. All the studies included in this survey were found in Scopus, Web of Science and IEEE Explorer and focus on automatic affect prediction in music. What is more, the literature written in a different language than English and not from the Computer Science field was excluded. Considering the time constraint of 10 weeks, an additional feasibility filtering was conducted. That is only the documents which mention common benchmark datasets for this task were included. The datasets were chosen after the research of related work and different literature reviews related to this topic. We were aware that it might include a bias correlated to dataset popularity. At the final stage of filtering, 113 records were chosen for data extraction from which 51 were used in the analysis due to time constraints and a single person conducting this research. After synthesising and analysing the extracted data we observed that the schemes can be distinguished between dimensional and categorical approaches and both are similarly popular. A rare but existing ARS are developed by combining both scheme types in a unique representation. What is more, there is no easily visible trend over time in the usage of certain schemes and only 66% of the reviewed studies support their choices with psychological theories. While conducting this research, we encountered limitations in terms of time and the research group, so we leave a gap for future improvements in this project.

Automatic affect prediction systems usually assume its underlying affect representation scheme (ARS). This systematic review aims to explore how different ARS are used for in affect prediction systems based on spoken input. The focus is only on the audio input from speakers. Various datasets for speech emotion recognition were also involved in the study to understand the motivation for certain (categorical or dimensional) schemes used for emotions. The basis, popularity, advantages and target affective states were investigated. We used Scopus and Web of Science to extract the papers, focusing on the systems in the field of Computer Science in English language. In summary, our exploration of affect representation schemes in Speech Emotion Recognition (SER) reveals a predominant focus on categorical representations of affect, particularly variations of Ekman's six basic emotions. Behavior and attitude, although rare, are also represented sometimes. Emotions like anger, happiness, and sadness receive the most attention, while the recognition of the neutral state as an emotional state remains controversial. Dimensional affect representation schemes are less common, possibly due to the difficulty in estimating valence solely from audio input. Researchers often combine multiple categorical schemes to accommodate different datasets used in SER systems, aligning the popularity of the schemes with the corresponding datasets. However, issues such as a lack of explanation for chosen categories, interchangeable use of terminology, and a weak psychological foundation for category selection pose challenges in achieving a comprehensive understanding of affect representation in SER research.

The objective of this report is to establish and present a machine learning model that effectively translates affect representation from emotional attributes such as arousal (passive versus active) and valence (negative versus positive) to dominance (weak versus strong). In the pursuit of this goal, various research questions are addressed. The paper outlines the process of dataset selection, ensuring appropriateness for the problem at hand. Subsequently, a comprehensive investigation into suitable evaluation methods for the developed model is conducted, providing well-reasoned justifications for the chosen approach. An additional research question focuses on assessing different machine learning approaches to determine the optimal performer. The motivation behind this translation lies in the recognition of the interdependence between these affect attributes, supported by both theoretical underpinnings and practical evidence. This contrasts with previous studies that have treated these dimensions as independent descriptors for representing emotions.

Spending time in front of screens has become an inescapable activity, which might be interrupted by unrelated external causes. While automatic approaches to identify mind-wandering (MW) have already been investigated, past research was done with self-reports or physiological data. This work explores automated detection utilizing solely facial expressions from Mementos data, which comes in the form of webcam recordings, where participants react to music videos. The recordings are annotated with labels indicating perceived MW. Video responses are turned into time series by first extracting facial characteristics, which are encoded with Facial Action Coding System (FACS). Temporal information is represented with 170 temporal features. Classification is conducted with support vector machines (SVM) through a data-level approach and an algorithm-level approach, first by synthesizing data and second by adding class weights to SVM. Both approaches are evaluated with metric scores insensitive to imbalanced data. On average,
results show that detection performs marginally better than by chance. However, the evaluation metric values vary across multiple classification runs, thus the prospect of using the Mementos dataset for automatic MW detection based on only facial expressions is not promising.

Mind wandering occurs when a person’s attention unintentionally shifts away from their current thought or task. Being able to automatically detect cases of mind wandering can assist applications with attention retention, and help people with maintaining focus. Many methods have been tested to deal with mind-wandering detection, but they are mainly conducted in controlled environments.
There also has been little study into the usefulness of learned features from neural networks. This paper is focused on showcasing the effectiveness of
using neural network-generated features as input for classification models. Specifically, using ResNet to generate features which are then used as input by supervised learning models for classification. These features and models were used to classify mind wandering in the Mementos data set, outside of a controlled environment or differently put as “In-the-wild“. The study shows that the extracted features could not be used to accurately detect mind wandering based on the F1-Score (Macro) measure. The results can be attributed to data imbalance, low amount of data, lack of dataset-tailored pre-processing operations, and indiscriminate features. To improve on the study, more data collection is advised and the usage of methods like re-sampling and data augmentation to deal with data imbalance. And lastly, experimentation with neural network training and transforming the data into a time series format to better represent the temporal information from the data.

Mind-wandering happens when one's current train of thought, related to a specific task, is interrupted, due to internal disconnected thoughts. This phenomenon is highly subjective, and its detection is really important due to the internal understanding of the human mind that can be obtained. Several methods have been used in order to detect mind-wandering, such as thought probes, self-reports, electrophysiological measures or even eye-based tracking methods, however the detection of mind-wandering solely from sound has not been researched about. Therefore, this study is going to investigate if automatic detection of mind-wandering through sound is feasible. In this work, this question is tackled through a machine learning approach, where a linear SVM model is trained through acoustic features. Two methods of oversampling are considered, due to the high data imbalance between the classes, and these two are evaluated and compared. The approach is evaluated through different metrics, such as recall, precision, F1-score and accuracy, but also a comparison with other techniques is done. Results of this work show that sound as its own is not a reliable way of automatically detecting mind-wandering. These results however, might be implementation specific, as the ground truth values of mind-wandering were created through the means of perceived mind-wandering, and the techniques of random oversampling and SMOTE were also used. These could be causes of unreliability of the research. Future work should should take this into consideration and also apply this approach to a different data set, to assess its feasibility.

Mind wandering is a phenomenon that is used to describe moments where a person's attention appears to shift away to something that is not related to the primary task, which can have a negative influence on the task performance. In this research, the aim is to create a viable algorithm that can automatically detect mind wandering based on eye movement from the Mementos data set. A method to automatically detect mind wandering could be used in online education in order to help students study more effectively, for example. The Mementos data set contains, unlike previous research, videos captured in an uncontrolled environment using inexpensive equipment. Features based on fixations and saccades were used to create two algorithms which were able to perform better than chance, having an average AOC-ROC of 0.63 and 0.59, as well as having an average F1 score of 0.046 and 0.041 compared to the chance-based model with an F1 score of 0.029.

The aim of this research is to discuss if it is possible or feasible enough to detect Mind-wandering of individuals using their hand and body movements from video recordings. The basis for this research is “Mementos”[9] data set, containing over 2000 recordings of people watching music videos. During experiment videos from data set were used to create software, that would automatically determine Mind-wandering. Results of the study have shown that body and hand movements are useful for detection of the phenomenon, however are not self-sufficient indicators for reliable detection. This study has found a reproducible methodology for automatic detection of Mind-wandering from pre-recorded videos, which is contributing towards exploration of this complex phenomenon in the field of Computer Science.