The advent of Virtual Reality (VR) technologies has begun a shift in communication between people and their interaction with 3D virtual environments. VR has great potential to provide high immersion to users, allowing designers to create vivid and impossible interactions. However, while software and technology play a crucial role in creating a VR experience, as designers we must understand how humans perceive these elements of sensory illusions in order to create experiences that are appropriately received and interpreted.
Recent efforts in “Social Virtual Reality” explore shared experiences and collaboration between users through remote interactions in virtual environments. One emerging concept is “Virtual Co-embodiment”, enabling two users to share a virtual character. This interaction fosters a unique multiplayer experience, promoting social co-ordination and collaborative user experiences. Co-embodiment achieves heightened levels of co-presence while still preserving a strong sense of agency and body ownership for both the users. The influence of feedback mechanisms on these factors is an important point of interest.
This project expands on this idea of co-embodiment by investigating how haptic feedback affects these factors between dyads performing shared perceptual activities. To examine these effects, an experiment was designed wherein pairs of participants in co-embodiment, performed reaching tasks with varying levels of control over the shared hand avatar, both with and without haptic feedback conditions. This was facilitated using a VR system that was tailor-made to meet these requirements. Objective measurements of their motion were collected during the interaction and subjective responses were recorded post-interaction.
The results showed that participants sense of agency was significantly lower in conditions where they received haptic feedback when their hand positions overlapped, compared to conditions where there was no haptic feedback. Participants made negative associations of the haptic feedback during the experiment as expressed in the post-experiment interviews, which could have affected their perceptions of agency. They also show significantly greater sense of agency during tasks where they shared a common target with their partner, while co-presence and embodiment levels were significantly higher in tasks where there were multiple targets. Participants also spontaneously adopted leader and follower roles during the interactions with different motion strategies to gain control over the shared avatar. These, along with other findings of the qualitative and quantitative analysis are compiled to extract insights to inform future research of this concept. Additionally, limitations of the study are discussed along with recommendations for further improvements to enhance this paradigm.

The focus of this project was the research and development of an affective social touch wearable. AffectiveAir uses pneumatic actuation on the shoulder to convey a library of haptic sensations. The goal was to overcome physical limitations in potential digital communication contexts, by a non-verbal, tactile, and possibly intimate social touch using pneumatic actuation. This would offer a way for individuals to connect, much like in face-to-face interactions

The embodiment of the prototype is the result of an iterative design process based on literature research, benchmarks, and user tests. The wearable actuator, designed to be worn on the back of the shoulder, is secured using an elastic band that attaches to the wearer’s pants. A custom-designed, thin polyurethane airbag with an integrated nozzle provides a soft, textile-integrated actuator solution. An external pneumatic control system controls the airflow, allowing for inflation under 1000- and deflation under 100 milliseconds. Effects of the actuator are monitored using an air pressure sensor and force sensitive resistor, offering controlled feedback and data logs for prototyping and user research.

Multiple user tests functioned as a means to explore and verify new designs throughout the project. Initial tests identified the shoulder as an effective and acceptable location for the feedback, and an airbag of 40x40mm was determined to provide the best balance between intensity and comfort for this specific location. Subsequent tests determined optimal pressure levels for pneumatic haptics on the shoulder and evaluated user responses to various pneumatic patterns.

Results revealed that it was challenging for participants to distinguish pressure levels from each other within the 0 to 500 mbar range. However, there were promising results in the general identification of increasing or decreasing pneumatic patterns using three pressure levels of 75, 200 and 500 mbar. ‘TripleShort’ received the highest identification rating at 85%, while ‘short staircase down’ received the lowest rating at 44%.

Patterns with a prolonged high pressure level at the end such as ‘heartbeat forward’ and ‘long staircase up’ received a slighly higher comfort and pleasantness rating compared to others, where the latter was rated the least exciting at lower speed. Overall ratings remained close to the median across tests with variable speed or pressure levels.

General user feedback on the prototype and haptic experience was positive, with participants noting the novelty and the sensation’s occasional resemblance to intimate human touch. Feedback from the interview also notes how context and social relationship status closely relates to the acceptability of receiving such feedback. The ergonomics of the prototype were considered satisfactory, allowing for adjustment to fit any body size and position the feedback on the back of the participant’s shoulder.

In conclusion, AffectiveAir demonstrated the potential for affective touch through a pneumatic shoulder wearable. It offers a library of identifiable and characterized pneumatic patterns and the ability to extend the possible actuations with adjustable parameters. The project’s outcomes suggest promising potential for further research using affective pneumatic haptics, applicable not only in mediated communication but also in other possible areas such as gaming and navigation.

Understanding how the end users, ordinary citizens, experience designed public spaces is fundamental in the field of urban design. Virtual Reality (VR) technologies significantly enhance the ease of evaluating public space designs by creating virtual simulations. This allows users to immerse themselves more deeply in the space compared to traditional screen-based presentations. However, the true experience of space design only occurs once it is physically constructed. Designers still face challenges in effectively investigating how citizens might interact with the future space. In evaluating public space designs, one obstacle is that users can only provide feedback based on individual experiences, lacking knowledge of how others have perceived or used the space, which contrasts with the real world situations where people experience and perceive public spaces with the association with how others’ experiences.

Trace visualizations have been used to analyze user behaviors. Visualizing traces of previous users unlocks possibilities of enhancing curiosity and meaningful exploration of the space, which could inspire more association about use scenarios and a deeper understanding of the spatial properties, and eventually benefit the evaluation outcomes. 

This project explored the influences of visualizing human traces on users’ explorative behaviors in virtual reality space. We started with conducting interviews with urban designers to understand the utilization of traces in the processes of public space design and their perspectives regarding implementing trace visualizations to support public space design evaluation. Insights from the interview informed the selection of traces for the experiment including user pathways, gathering situation, full-body motion, and space atmosphere. Subsequently, we integrated trace visualizations into a VR shopping street and conducted the experiment with 20 participants to evaluate the impacts of each trace visualization on user exploration. The outcomes from interviews and operational activities undertaken by participants revealed exploration patterns associated with each type of the trace visualizations and related findings concerning user perceptions of the trace visualizations. The project concluded with the implications for implementing trace visualizations in VR environments and the limitations of the research.

Emotion Recognition is one of the vastly studied areas of affective computing. Attempts have been made to design emotion recognition systems for everyday settings. The ubiquitous nature of Intelligent voice assistants (IVAs) in households, make them a great anchor for the introduction of emotion recognition technology to consumers. The existing systems lack such pipelines and rely on dictionary-based architectures in their design. Further, these systems lack conversational properties and are merely an extension of information retrieval engines.

In this setting, we propose to introduce and develop emotion recognition pipelines that are suited to the interactions, common with these IVAs. To augment the existing emotion recognition pipelines which rely on audio information, we look at physiological information derived from wearables. Our proposed model uses multimodal embeddings with a Siamese Network to achieve the task of emotion recognition from a few samples. Physiological signals of blood volume pulse (BVP) and electrodermal activity (EDA) are used as additional input embeddings to two audio embeddings arising from the speech samples. We employ the state-of-the-art training schedules for Siamese Networks, which use a very limited amount of training on support datasets via sample pair comparisons. The performance of the model is evaluated using weighted binary accuracy and f1 scores.

The proposed model is applied on two datasets that denote two unique experimental settings - the K-EmoCon dataset and RECOLA dataset. We demonstrate an improvement in the state-of-the-art accuracy with the K-EmoCon dataset with accuracies of 63.97% and 66.91% on arousal and valence dimensions respectively. Further, on the RECOLA dataset, the model performs moderately well with 53.81% and 53.87% respectively for arousal and valence dimensions. In addition to this, we present a study of the effects of variation of available support set for training from the dataset. We make some salient observations for these experiments across individual participants and also identify how the label distributions affect the performance of the model. Further, we investigate the impact of real-world noise samples from the DEMAND dataset on the two datasets. We observe that the proposed model is robust and performs sustainingly well even in the presence of imputed noise.

Social VR is the application of virtual reality that supports remote social interaction in virtual spaces. Users communicate and interact with others in the social VR environment through avatars, which are virtual anthropomorphic characters that aim to represent humans in virtual worlds. In addition, the development of the HMD and commercially available motion capture systems enable the avatars in the virtual environment to detect and reflect the real-time motions, even facial expressions of people. However, the avatars still lack an indication of biofeedback - e.g., body temperature, breathing, heart rate, muscle contraction -, which serves as social cues for communication in reality. While some features, for example, emojis, supports users to express their feeling or emotions for richer communication, the missing information often results in miscommunication in the virtual space. It remains a barrier to a fully immersed experience in the social VR space. This project proposes a concept of visualizing biosignals of the avatars in the social virtual reality space for a richer-level interaction in virtual reality. With the technologies available to capture and reflect accurate biofeedback in real-time, we would like to explore ways and possibilities to map the bio states of the users in reality to avatars in the virtual world. The project starts with conducting user researches to understand the current user behaviors in the social VR spaces and their perspectives on sharing biosignals. Based on the requirements gathered from the user study, the scope of the project is narrowed down to a ‘watching entertainment’ scenario, and the ways to visualize biosignals on avatars were explored through a co-design session with designers. After that, four biosignal visualization techniques in two biosignals - heart rate and breathing rate - are prototyped under the VR jazz bar setting. Finally, the user study is conducted with 16 pairs (32 participants in total) to test and compare the effects of each biosignal visualization technique in watching entertainment scenarios with a companion. As a result, the embodied visualizations are the most understandable and least distracting visualization method among the four methods. Furthermore, the limitations of the research, recommendations on biosignal visualizations, and recommendations on conducting design research are provided.

Facial emotion recognition from thermal images has gained more attention in recent years. Thermal cameras capture the heat emitted by objects and therefore thermal images are not sensitive to illumination changes. Furthermore, changes in temperature can indicate emotions and it is harder for humans to fake emotions in front of a thermal camera. However, a limitation is that, thermal cameras that capture high-resolution images are expensive, and cheaper thermal cameras often capture images with a low-resolution and/or contaminated with noise and blur. Besides, low-resolution thermal images can also arise when images are captured from a far distance or from moving persons. When using these low-resolution thermal images for facial emotion recognition this can negatively influence the emotion classification accuracy. To tackle the problem of low-resolution thermal facial expression images, super-resolution can be used. In this exploratory work, we propose the Thermal Face Super-Resolution Network (TFSRNet) and the Thermal Face Super-Resolution Generative Adversarial Network (TFSRGAN) to recover high-resolution thermal facial expression images from low-resolution thermal facial expression images, with the goal to use the super-resolved images for thermal facial emotion recognition. The architecture TFSRNet is optimized to minimize the mean squared error (MSE), which results in images with a high peak signal-to-noise ratio (PSNR). However, these images often contain an unsatisfying perceptual quality. To generate high-resolution images with a high perceptual quality we propose TFSRGAN. Both architectures use facial prior knowledge, such as facial landmark heatmaps and parsing maps, to enhance low-resolution thermal facial expression images. To emphasize the most important parts of each facial expression and to suppress irrelevant facial parts, we integrate the Convolutional Block Attention Module (CBAM) in both super-resolution architectures. The proposed super-resolution architectures are used to enhance low-resolution thermal facial expression images, which are obtained with three different degradation models, namely bi-cubic down-sampling (BI) on scale x2, x3 and x4, blurring followed by bi-cubic down-sampling (BD) on scale x3 and bi-cubic down-sampling followed by adding noise (DN) on scale x3. With an ablation study, the effectiveness of using facial prior knowledge and the attention mechanism CBAM for thermal super-resolution is shown. When using facial prior knowledge and the attention mechanism CBAM, the image quality of the super-resolved images improves. Furthermore, experiments show that images enhanced by TFSRNet outperform bi-cubic interpolated images, for degradation models BI x4, BD x3 and DN x3. Using these super-resolved images for thermal facial emotion recognition also leads to an increase of the emotion classification accuracy. In addition, images enhanced by TFSRGAN outperform bi-cubic interpolated images for degradation model DN x3. Although, this an exploratory work containing limitations, the experiments show the effectiveness of using facial prior knowledge and the attention mechanism CBAM for thermal facial expression super-resolution. In addition, thermal face super-resolution shows promising results for thermal facial emotion recognition where future work can build upon.

The project aims at exploring tangible ways to communicate emotion. To narrow down the scope, I lay the focus on emotion communication in remote settings, a context when nonverbal communication is usually limited. From interviews, the neutral expression is found to be the main factor hindering a sufficient understanding of others' emotional state. Therefore, the scope is narrowed down to a topic related to the real-world problem: Contributing to the emotion interpretation from the neutral verbal expression.
While verbal communication cannot convey emotion sufficiently, tangible modalities like haptics, thermal feedback are promising extra channels to explore. Communicating affection with the physical stimulus is not only supported by theoretical studies like embodying emotion theory but also practiced through prototyping and experimenting from HCI filed. Within this project, thermal stimulus, a modality highly related to emotional experience without causing privacy issue, is selected for the study. Thus, the research can be further framed as: How thermal stimulus contribute to emotion interpretation from the neutral verbal expression? Considering the application perspective and the requirement about wearables from CWI, the thermal stimulus should be designed as wearables. The research is conducted in the path of the controlled experiment to generate solid findings. Voice message, a medium less explored in the area of remote communication, is chosen as the case for study. Therefore, the final research question is formulated as follows: How can wearable thermal display contribute to emotion interpretation from the neutrally spoken voice messages?
The project follows the path of research through design. First, through literature review and quick user test, a pre-study leads to some initial decisions for the experiment: 1) using the upper chest as the body location. 2) using woven silk as the contact medium. 3) setting four thermal stimuli (high intensity warm 38°C, low intensity warm 35°C, high intensity cold 26°C, low intensity cold 29°C).
Afterward, the first experiment is conducted to find the recognizable and acceptable on-body thermal stimulus with a suitable contact medium (skin or fabric) for the second experiment. The study with 12 participants collects the ratings in terms of subjective intensity and comfort. The findings indicate that low intensities cannot be used if the thermal stimuli should be detectable for most people. For the cool stimuli, the intensity is decreased from -6°C to -4°C to ensure participant comfort. Besides, the fabric is not used for the second experiment because of its delay effect on thermal sensation.
Between the two experiments, a study is conducted to validate that the neutral tone can make a normal voice message perceived more neutral. Twelve neutrally spoken voice stimuli are generated by an AI voice generator and compared with the original normally spoken voices (from a validated database) in terms of valence and arousal. The result shows that the neutral expression does make a sentence with either positive or negative content perceived more neutral.
The second experiment is to explore how thermal stimulus can affect emotion perception from nuetrally spoken voice messages. Participants listen to eight voice messages and experience the thermal stimuli at the same time. Afterward, they give their ratings in terms of valence and arousal from the voice messages. Each voice message is paired with three thermal conditions (warm, cold, baseline). The result shows that warm stimulus augments the perceived positiveness of positive messages while cold stimulus augments the negativeness of negative messages.
However, augmenting the emotion perception doesn't necessarily mean the accuracy of emotion perception. The contribution and limitation of the project are discussed in the end.

Sleep is a natural state of our mind and body during which our muscles heal and our memories are consolidated. It is such a habitual phenomenon that we have been viewing it as another ordinary task in our day-to-day life. However, owing to the current fast-paced, technology-driven generation, we are letting ourselves be sleep-deprived, giving way to serious health concerns such as depression, insomnia, restlessness, apnea and Alzheimer’s. Polysomnography (PSG) studies are used for diagnosing and treating sleep-related disorders. Although the PSG studies are considered as the gold standard, they are obtrusive and do not allow for long-term monitoring. Various wearables have been manufactured to help people monitor their sleep-health. However, these devices have been shown to be inaccurate.

The ubiquitous sensor technology employed by the wearables provides large volumes of data, recorded in the most natural setting of the user. There is an opportunity to make use of the highly available sensor data to model a sleep scoring system that could help individuals monitor their sleep-health from the comfort of their home. In this thesis, we aim to alleviate this problem by attempting to bridge the gap between the highly accurate but obtrusive medical diagnosis (PSG) and the non-intrusive yet inaccurate wearables.

In this work, we propose DeepSleep, a deep neural net-based sleep classification model using an unobtrusive BCG-based heart sensor signal. Our proposed model’s architecture uses the combination of CNN and LSTM layers to perform self-feature extraction and sequential learning respectively. We show that our model can classify sleep stages with a mean f1-score of 74% using the BCG signal. We employ a 2-phase training strategy to build a pre-trained model to tackle the limited dataset size and test the transferability of the model on other types of heart-signal. With an average classification accuracy of 82% and 63% using ECG and PPG based heart signal respectively, we show that our pre-trained model can be used in the transfer learning setting as well. Lastly, with the help of a user study of 16 subjects, we show that the objective sleep quality metrics correlate with the perceived sleep quality reported by the subjects with a correlation score of 푟 = 0.43.

Although our proposed model’s performance is not yet comparable to the medical standards, we show that it is possible to monitor our sleep-health using the wearable signals with the least domain knowledge and preprocessing techniques. The prediction and performance of our DeepSleep model show that it is able to learn the biological rules of sleep wherein it always follows a Deep or REM stage with a transitional Light stage. Our model treats the classification problem sequentially, thus, identifying important sleep parameters like the onset of sleep cycles and time spent in different sleep stages which are time-dependent factors. Furthermore, our user study, conducted using the SATED questionnaire, provides an insight into the difference in the user’s perceived sleep quality and model’s estimation. It shows that an automated classification system needs to incorporate various external factors such as environmental and ambient conditions to be able to strongly correlate with the perceived or subjective quality. We further discuss the future research gaps and opportunities that could improve the model’s performance and also extend it to other domains like irregular heart-beat and apnea detection. We consider this work to be a starting point for research into sleep and heart health using non-intrusive wearable sensors and deep neural network-based architectures.