Affect Representation Schemes Used in Affective Video Content Analysis

Abstract

Affective Video Content Analysis aims to automatically analyze the intensity and type of affect (emotion or feeling) that are contained in a video and are expected to arise in users while watching that video. This study aims to provide a systematic overview of various affect representation schemes utilised by researchers in the field of Affective Video Content Analysis and the reasons behind their choice. The main objectives of the study were to investigate the diversity of affect representation scheme types, their popularity over time, the basis of their selection, and the relationship between input data sources, in terms of direct and implicit analysis, and scheme types. Following the PRISMA guidelines to conduct a systematic literature review, a total of 45 papers were included in the study which were original journals and conference proceedings in the field of Affective Video Content Analysis and that were related only to Affective Movie Content Analysis published in English after 2008. Papers concerning Video Emotion Recognition were excluded from the review. The findings reveal that dimensional, categorical, and combined approaches are commonly used in this field, with the dimensional approach based on valence and arousal being the most prevalent. However, there is no significant trend in the popularity of affect representation scheme types over time. The study highlights the lack of clear motivation and explicit justifications for scheme selection, emphasizing the need for transparency and the inclusion of psychological theories as a basis for scheme choices. Additionally, the study found that audiovisual data was the most commonly used input compared to physiological signals and visible behavioural data. The study acknowledges its limitations, including time constraints and single researcher involvement, and suggests allocating more time and involving multidisciplinary teams for comprehensive insights.