Many studies have indicated that predicting users’ perception of visual quality depends on various factors other than artifact visibility alone, such as viewing environment, social context, or user personality. Exploiting information on these factors, when applicable, can improve users’ quality of experience while saving resources. In this paper, we improve the performance of existing no-reference image quality metrics (NR-IQM) using image semantic information (scene and object categories), building on our previous findings that image scene and object categories influence user judgment of visual quality. We show that adding scene category features, object category features, or the combination of both to perceptual quality features results in significantly higher correlation with user judgment of visual quality. We also contribute a new publicly available image quality dataset which provides subjective scores on images that cover a wide range of scene and object category evenly. As most public image quality datasets so far span limited semantic categories, this new dataset opens new possibilities to further explore image semantics and quality of experience.@en

Multimedia systems are typically optimized in a way that maximizes users’ satisfaction of using the systems/services. This user satisfaction is what is commonly referred to as Quality of Experience (QoE). For visual media, such as images and videos, the optimization of QoE has meant reducing the visibility of artifacts (e.g. noise or other disturbing factors) in the visual media. This is based on the assumption that the sole appearance of artifacts would disrupt the whole visual experience, in a world where media weremostly consumed passively, and in well defined contexts (e.g., TV broadcasts). Nowadays, the way users experience visual media has changed, thanks to the diffusion of mobile, interactive, immersive, and on-demand technology. Media are now consumed in many different contexts, for example, in the interactive and customizable contexts of social media, or in the immersive contexts of virtual and augmented reality. As consequence of these developments, a user’s visual QoE is no longer determined solely on the appearance of artifacts, but also by factors relevant to the viewing context. This thesis brings in new insights in modeling and automatically assessing users’ visual QoE in view of the developments above. The thesis starts with looking into subjective methodologies for QoE assessments, and continues with developing objective quality metrics that incorporate QoE influencing factors to improve state-of-the-art metrics. Developing reliable and accurate objective metrics to automatically assess users’ visual QoE requires subjective data that are reliable as well. This thesis argues that existing methodologies for collecting subjective data might not be reliable when used to evaluate QoE factors that are highly subjective, or that are new to the research community. Highly subjective quantities may yield different conclusions across experiments. As for new types of media, they often bring the uncertainty on how to evaluate them. Two studies are then presented in this regard. The first study considers the assessment of image aesthetic appeal, as one example of a highly subjective quantity. A large scale study was conducted to compare the use of different subjective methodologies to collect aesthetic appeal data, and some ways to measure the data reliably were proposed. The second study considers the assessment of point cloud quality, as one example of a new type of media (i.e. immersive media). The study explores quantitative and qualitative approaches to understand the way users judge point cloud images. Following the studies on subjective QoE assessments, two studies on objective QoE metrics are presented in this thesis. Despite existing efforts to model the influence of different factors on visual QoE, limited work have proposed to incorporate these factors into existing objective qualitymetrics to improve state-of-the-art. The first study on objective QoE metrics in this thesis investigates the influence of image content/semantic categories (i.e. scene and object categories) on visual QoE, and proposes to include semantic category features in objective image quality metrics. The proposed approach shows improvement from state-of-the-art in predicting image quality. The next study on objective quality metrics investigates new QoE influencing factors for point cloud images, and proposes to incorporate these into an objective quality metric for point cloud images. The results of the studies presented in this thesis show how existing subjective methodologies could yield reliable aesthetic appeal data, and explore point cloud QoE influencing factors. Moreover, the results show that incorporating new QoE influencing factors into objective image quality metrics could improve state-of-the-art performance in predicting users’ QoE. At the end of this thesis, some recommendations are given for future research following up the findings in this thesis. @en

As Virtual Reality (VR) applications gain more momentum recently, the social and communication aspects of VR experiences become more relevant. In this paper, we present some initial results of understanding the type of applications and factors that users would find relevant for Social VR. We conducted a study involving 91 participants, and identified 4 key use cases for Social VR: video conferencing, education, gaming and watching movies. Further, we identified 2 important factors for such experiences: interacting within the experience, and enjoying the experience. Our results serve as an initial step before performing more detailed studies on the functional requirements for specific Social VR applications. We also discuss the necessary research to fill in current technological gaps in order to move Social VR experiences forward.

@en

The driving force behind digital crowdsourcing are its workers: working, hidden behind the scenes, churning out data in experiments, participating in research studies, completing little tasks to accomplish HITs online. Understanding workers and crowdwork better is therefore key to develop a more effective and fair use of crowdsourcing for research. This chapter attempts to help develop an understanding of the various aspects of the crowd by drawing parallels between workers of different platforms (AMT, Microworkers and Crowdee) through quantitative and qualitative analysis of current and newly collected data. A picture of the crowd is drawn by uncovering their motivations, workplaces, skills and infrastructure, issues and perspectives about the design of microtasks, the employers and the microtask-based platforms. Legal and ethical perspectives on crowdwork are also discussed, and online resources are reviewed that researchers can use as a primer to employ crowdworkers in an ethical and fair way. The chapter provides information, a review of internationally recognised ethical principles and practical advice to those who would like to use crowdsourcing for experiments and to carry out research studies as an informed researcher and crowd employer.

@en

The driving force behind digital crowdsourcing are its workers: working, hidden behind the scenes, churning out data in experiments, participating in research studies, completing little tasks to accomplish HITs online. Understanding workers and crowdwork better is therefore key to develop a more effective and fair use of crowdsourcing for research. This chapter attempts to help develop an understanding of the various aspects of the crowd by drawing parallels between workers of different platforms (AMT, Microworkers and Crowdee) through quantitative and qualitative analysis of current and newly collected data. A picture of the crowd is drawn by uncovering their motivations, workplaces, skills and infrastructure, issues and perspectives about the design of microtasks, the employers and the microtask-based platforms. Legal and ethical perspectives on crowdwork are also discussed, and online resources are reviewed that researchers can use as a primer to employ crowdworkers in an ethical and fair way. The chapter provides information, a review of internationally recognised ethical principles and practical advice to those who would like to use crowdsourcing for experiments and to carry out research studies as an informed researcher and crowd employer.

@en

We hypothesize that the semantics of image content affects how humans judge the perceptual quality of images. The recognition of image content has been shown to be processed within the first 500 ms of observation (and mostly in a pre-attentive stage). We look at whether or not participants are also able to detect impairments and judge their annoyance at early attentive stages. As the presence of impairments may slow down the early semantic recognition process, we investigate whether or not different semantic content impacts people’s judgment of image quality. Our results show that participants do recognize image content despite the presence of impairments even at very early stages of vision (within the first fixation). In addition, we show that semantic categories have an influence on people’s detection of image impairments at early attentive stages. People seem to be able to correctly detect very obvious impairments within one fixation, but more subtle impairments are not perceived. Finally, we show that people are more tolerant toward impairments on images portraying outdoor scenes than images portraying indoor scenes; additionally, users seem to be more critical toward images containing animate objects (humans or animals) in the region of interest compared with those with inanimate objects.@en

Recognizing what makes an image aesthetically pleasing is crucial to the effectiveness of many multimedia systems. Several works have attempted to build image aesthetic appeal predictors, and created their own set of ground truth data for the purpose, either by using rated images from photo sharing websites, or by asking a pool of users to rate images in lab or crowdsourcing experiments. Literature has shown that the way these experiments are conducted can influence their results: poor experimental setup can result in poorly reliable outcomes (i.e., highly imprecise aesthetic appeal measures). A question then arises whether the different choices made to collect ground truth of aesthetic appeal data are appropriate. In this paper, we propose a systematic study that looks into how different experimental environments and rating scales used to collect image aesthetic appeal ground truth data influence the reliability and repeatability of aesthetic appeal assessments. Our findings show that discrete and continuous scales with five-point absolute category rating labels yield more reliable results, with the continuous scale being more reliable for abstract images. We also show that image aesthetic appeal assessments could be repeatable across different experimental environments (i.e., lab and crowdsourcing). We finally formulate concrete recommendations to guide the collection of large sets of ground truth data for training models of aesthetic appeal appreciation.@en