Visual impairment affects over 2.2 billion individuals globally, emphasizing the critical need for effective assistive technologies. This work focuses on developing a video captioning model explicitly tailored for visually impaired users, leveraging advancements in deep learning techniques. Video captioning involves converting video frames into textual descriptions, effectively bridging the domains of computer vision (CV) and natural language processing (NLP). We surveyed young visually impaired individuals from the Visio organization, who provided key insights into the design of our model.
We enhance the existing S2VT model by modifying its temporal attention mechanism to improve the recognition of visual surroundings, addressing the unique challenges visually impaired individuals face.
This research explores critical questions surrounding the model's sensitivity to actions, the readability of generated captions, and methods for latency reduction. To evaluate the model's effectiveness, we implement readability metrics—an approach not previously utilized in video captioning assessments. Our findings contribute to enhancing accessibility and independence for visually impaired individuals through advanced video captioning solutions.

Training deep learning models for time-series prediction of a target population often requires a substantial amount of training data, which may not be readily available. This work addresses the challenge of leveraging multiple related sources of time series data in the same feature space to improve the prediction performance of a deep learning model for a target population. Specifically, we focus on a scenario where the target dataset, representing the desired target population, is underrepresented, while the source datasets consist of mismatched populations that are sufficiently representative for training a deep learning model. In this study, we explore state-of-the-art techniques, including transfer learning, ensemble learning, and domain adaptation to leverage source datasets towards a target population using real-world medical data. Additionally, we investigate the use of model performance-derived baselines as a heuristic to quantify the magnitude of the distribution mismatch between a source(s) and a target. Our results demonstrate that a set of well-defined baselines can effectively quantify the distribution mismatch and provide insights into the choice of leveraging technique for a given mismatch scenario. Furthermore, our results show that all state-of-the-art techniques can be employed to leverage related source datasets towards the target, though the performance of these techniques varies depending on the characteristics of the distribution mismatch. Eventually, we discuss the applicability of this research to new scenarios, along with avenues for future research.

Moral values influence humans in decision-making. Pluralist moral philosophers argue that human morality can be represented by a finite number of moral values, respecting the differences in moral views. Recent advancements in NLP show that language models retain a discernible level of knowledge in deontological ethics and moral norms of society. However, a model which can only decide either right or wrong cannot fully understand the diverse moral perspectives of humans.

We propose a moral sentence embedding space, which can encompass moral differences, through the state-of-the-art Contrastive Learning framework. We evaluate the moral embedding space both intrinsically and extrinsically via three tasks: classification, moral similarity, and visual analysis. We show that our moral embedding space understands the characteristics of each moral value. Our results also highlight that moral rhetoric is seldom explicit in the text, emphasizing the necessity of additional information such as moral labels.

Artificial social agents (ASAs) are systems designed to interact with humans in a socially intelligent manner. As the field of robotics is rapidly advancing, some studies focused on creating more effective agents by analysing how people perceive them. However, culture affects people’s perception of ASAs. So, cultural aspects must be examined in order to create more effective ASAs. This study aims to contribute to the research of cultural influence on ASAs by discovering the cultural similarities and differences in the perception of ASAs from Dutch and Chinese speakers. An English questionnaire has been made to evaluate ASAs and also has been translated into Chinese. In this study, the questionnaire is translated into Dutch and validated using both the original and the translated questionnaire. While assessing the translation, data is gathered from Dutch speakers on a human-ASA interaction. This data is then compared to the previously collected data from the Chinese translation study by calculating their mean, standard deviation and t-distribution of the differences between both cultural groups. On an item level, the results show a satisfactory level of correlation (ICC M = 0.67, SD = 0.12, range [0.36, 0.92]). On a construct/dimension level, results show a good level of correlation (ICC M = 0.80, SD = 0.11, range [0.52, 0.93]). Correction values for the translation have been advised for converting item scores between the two questionnaires. Cultural differences have been found and reported between Dutch and Chinese speakers, which could be used in future research on creating more effective ASAs.

The Artificial-Social-Agent (ASA) questionnaire is an instrument for rating Human-ASA interactions. The questionnaire was built to make evaluation results more comparable between different agents and to make findings more generalizable. The ASA questionnaire was designed in English, which limits its usage to people who have a firm grasp of the English language, so translating the questionnaire into different languages allows for the inclusion of other people and cultures, as well as opening up the possibility of comparing different cultures. A previous study performed a translation of the ASA questionnaire into Chinese, and this repeats a similar process by translating the questionnaire into Dutch. This research project employs a survey-based approach, utilizing the Prolific and Qualtrics platforms to collect valuable insights and responses from bilingual English and Dutch participants. Participants watch a video depicting an ASA-human interaction and then complete a questionnaire designed to assess their perceptions. The main research question focuses on identifying the cultural variations in the perceptions of ASAs. By addressing this question, this research contributes to the existing body of knowledge on ASAs and culture. The methodology involves translating and refining the questionnaire, followed by data collection and analysis using Intraclass correlation coefficients (ICCs) to measure agreement between the English and Dutch versions of the questionnaire. The final evaluation and analysis includes a deeper statistical analysis, and a comparison between Dutch-English bilingual participants, and mixed-English participants.

The work presented in this thesis investigates the creation of virtual sound sources in a room equipped with a limited number of loudspeakers. This limited number of loudspeakers is typical for consumer loudspeaker systems. Ideally, these systems can provide a listening experience in which localisation cues are completely present. However, in current systems, this is not the case. The limited number of loudspeakers make it impractical to produce a physically accurate sound field. A possible solution is to create a perceptually accurate sound field instead. In this work, a step towards an algorithm which can do so is presented. The developed algorithm requires knowledge of the listener placement, the room, and the loudspeaker placement. Spatial weighting is used to construct a region in which the acoustic en-ergy should be large and a region in which the acoustic energy should be limited. The loudspeaker playback signals are obtained by maximising the energy ratio between these regions, while at the same time ensuring that the perceptual difference between the received audio and reference audio remains limited. The algorithm employs a convex optimisation problem to facilitate efficiently solving for the playback signals. Concretely, six convex optimisation problems are proposed with somewhat increasing complexity and different weighting matrices. For each of the optimisation problems, the proposed algorithm is compared against a simple amplitude panning algorithm and a nearest neighbour algorithm. It is found that none of the considered algorithms is clearly preferred over the others in terms of the considered evaluation metrics. A major limitation of the presented work is that the evaluation metrics do not explicitly test for localisation accuracy. In future work, this should be investigated by including subjective tests.

Over the past decade, spatial audio awareness evolved into an in-demand feature in audio entertainment. The addition of sound source locations to, for instance, movies or music adds a level of auditory envelopment and spatial awareness to the audio experience. Expensive setups present in, for instance, cinema's, are able to create this envelopment by means of a large set of loudspeakers with which the desired sound fields are created. Creating this spatial envelopment in practical consumer living rooms or home cinema setups proves to be a more challenging task due to the impractical amount of loudspeakers required. To create the audio envelopment with a small amount of loudspeakers, crosstalk cancellation can be used. Crosstalk cancellation as posed in literature is, however, not robust enough to be used in practical appliances. The main cause of these bad characteristics is the objective cost function it optimizes which results in an ill-posed problem. In this thesis, the crosstalk cancellation problem is relaxed by aiming for perceptually sufficient results instead of aiming for objectively optimal results. The human auditory system has its limitations in both the perception of audio and localization of audio sources. Exploiting the limitations of the auditory system generates mathematical freedom that can be used to construct a more robust and stable crosstalk cancellation algorithm. This thesis provides the first steps towards the incorporation of audio perception in the domain of crosstalk cancellation and audio envelopment with a limited amount of loudspeakers.

The Socially Perceptive Computing Lab (SPCL) at Delft University of Technology has developed a device called the Midge. The aim of this device is to record data of social interactions at conferences. This paper aims to characterise how battery life is affected by different sensor settings on the Midge as well as how much data is generated in a given period of time. Based on the findings from the performed tests, formulae for both run time and data generation were devised.

Detecting social interactions through wireless wearable Bluetooth devices is increasing in popularity. Devices use the signal strength to other detected devices to estimate the proximity between people and group them together based on the Dominant set algorithm. Dominant sets are a maximal clique of nodes with an edge-weight based on the affinity between the nodes. Nevertheless, the signal is heavily influenced by external factors, which increase in an crowded environment. This paper introduces three different noise reduction filters that try to detect the kind of noise and therefore improve the detection of surrounded devices. Further, this paper looks at the overall impact of proximity on the resulting RSSI values. Knowing this relationship helps to normalize the values and therefore eliminates the need to apply noise reduction. Using a dataset of 48 sensors recorded in a conference setting with a specific designed sensor the low frequency pass filter gets an accuracy score of 81.8 % with a cut-off frequency of 0.07 Hz. It performs best when considering a conversation window of 20 seconds. Here, only 2/3 of the detected groups has to coincide with the actual formed group at a specific timestamp. Furthermore, the orientation of the participants to each other has heavy influence on the resulting RSSI values and therefore a normalization based on only proximity cannot be done.

The Midge is a wearable badge created by the Socially Perceptive Computing Lab, Pattern Recognition and Bioinformatics group of the Delft University of Technology, with as goal to analyse human behaviour. The badge has a digital motion processor (DMP) that can determine its orientation. This DMP makes use of an inertial measurement unit (IMU), that houses an accelerometer, a gyroscope and a magnetometer, to calculate its movement in a 3D-space. For both of the accelerometer and gyroscope the Full Scale Range (FSR) can be changed, in addition to the frequency. In this paper, the effects of both elements are analysed to determine if they influence the accuracy of the data gathered. The results show that the changing the FSR does not influence the accuracy of neither the two sensors nor the performance of the DMP. On the other hand, it was found that changing the frequency does influence the performance of the Midge. Even though the frequency did not affect the measurements of the accelerometer and gyroscope directly, the performance of the DMP was affected. The DMP performed best with a frequency of 150 Hz. Using a higher frequency also captured local extremes and turning points from the sensors and the interpreted orientation more precisely.

In our daily life people encounter many social interactions, for example in the supermarket, at work and in schools. Currently the most reliable way to find social interactions in groups, is to manually annotate the data. Manual annotation takes a lot of time and human resources and as the information stream goes faster and faster, the manual labour cannot keep up. Therefore an automated program to replace this is desirable. One method to automate this, is by looking at the proximity of people, which has been done by Dikker [1]. The results look promising, however they are sensitive to errors. The F1-score (equation 3) is only 0.625 and the precision (equation 5) is 0.696. This means that there are still a lot of false positives. With orientation data one can determine if the people in question are facing each other before they are assigned to be in social interaction. With a view frustum (cone shape) it can be determined who are facing at each other by checking if the cones overlap. This cone can take different sizes, making the view area bigger or smaller. Because the absolute rotation were only used, it was used to find out who cannot face each other. Using the proximity results as a baseline, the impossible orientations were removed. With the goal to reduce the amount of false positives. From the results it can be concluded that a small cone will perform worse compared to the baseline. The bigger the angle of the cone becomes, the closer it gets to the results from Dikker [1]. However, in none of the results it turns out that using absolute orientations improves the current work significantly.

Smoking is a leading risk factor negatively impacting the health of people, not only those partaking in it first-hand, but also to those around them. Different methods are available to assist people with quitting smoking, with various degrees of effectiveness. Researchers developing smoking cessation approaches would like to have controlled environments to test how effective they are before offering them as viable options. Virtual reality has been demonstrated to be an efficacious tool for facilitating the presentation of cues aimed at eliciting smoking cravings in the lab. Addiction, however, is a complex matter involving different parts of the brain, and how or when conditioned responses causing smoking cravings are activated depend on the individual in question. There are strong indications that personalization of smoking cues, or in other words using elements relevant in the addiction model of an individual, may elicit higher or at least consistent smoking cravings. The general aim is that developing smoking cessation approaches targeting those most relevant elements may have higher ecological validity and therefore be more effective in assisting people both with quitting smoking and maintaining that behavior change. To assist researchers with testing their smoking cessation approaches using a virtual coach, we have created a system enabling the presentation of virtual environments and facilitating communication between researchers and participants while the latter are viewing them. We have evaluated the effectiveness of our system in eliciting a familiar experience, which we posit is a major component tied to smoking cravings. Our results showed that personalized virtual environments elicited a more familiar experience than non-personalized ones. We also examined the usability of our developed user interface, as well as the sense of presence elicited by our system, both of which received positive scores.

Recent advancements in quantification of repair outcomes of CRISPR-Cas9 mediated double-stranded DNA breaks (DSBs) have allowed for the use of machine learning for predicting the frequencies of these repair outcomes. Local DNA sequence context influences the frequencies of mutations that arise when DNA gets repaired after it is targeted by CRISPR (CRISPR outcomes). Contemporary models exploit this and can predict what the frequencies are of CRISPR outcomes at predetermined genomic loci. Predictions of such models are reasonably precise, but there may be opportunities for improvement in how the DNA sequence context is leveraged for making predictions. Some models only utilize a set of hand-crafted features, limiting the available information for the model. Other models do utilize broader sequence context but disregard sequence order or only predict a limited set of outcome classes. In this work we present an attention-based deep learning model that uses DNA sequence context to make fine-grained CRISPR outcome predictions. We present a custom input embedding for representing DSB repair outcomes and we expand on existing methods for analyzing attention-based models.

The Midge is a sensor device developed by the Socially Perceptive Computing Lab (SPCL) at Delft University of Technology (TU Delft). This device is used to monitor human behaviour in social settings using several sensors. In this paper, the accuracy of the Inertial Measurement Unit (IMU) chip used in the Midge sensor package was evaluated. The IMU is responsible for sensing motion in multiple directions using an accelerometer, gyroscope and magnetometer. This experiment was performed by comparing several Midge devices with each other as well as comparing them with a modern smartphone. First will be explained how the control software was updated to run on modern hardware (computers) and the work that was done to reliably convert the generated binary data to readable data (parsing). Then the process of creating the test setup, performing the tests and analyzing the data will be explained.

The goal for this paper is to find out what the smart badge provided by the Social Perceptive Computive Lab (SPCL) group is and what it contains. The sensors that are used in the smart badge are the Accelerometer, Gyroscope and Magnetometer. The main question of this paper is ”What is the benefit of using full 9-DOF IMU data in predicting speaking status, as opposed to using only accelerometer signals?”. The three senors all contribute in their own way and complement each other to give an estimate about the speaking status. The ability to estimate the speaking status using the smart badge opens up the potential for analyzing more about the social aspects of people without the need to record what they are saying.

Parkinson’s Disease (PD), Essential tremor (ET), and dystonia are movement disorders often misdiagnosed as one another and commonly present tremor as one of their motor symptoms. Rates of misdiagnosis between 30 and 50% of ET patients have been reported, where dystonia and PD are the most common missed diagnoses. Additionally, up to 50% of dystonia cases are misdiagnosed/under-diagnosed at their first encounter. Misdiagnosis rates up to 34% are reported for PD. Although similar tremor behaviors between the mentioned disorders lead to substantial misdiagnosis rates and, consequently, subpar care, tremorous signal acquired via wearable sensors can be used to discriminate between PD, ET, and dystonia patients. This study aims to develop three proofs-of-concept, accelerometer-based algorithms to assist medical doctors with decision-making in unclear diagnostic scenarios.

We live in a world where much of our interactions with the environment around us depend on us being physically close to them. For instance, we have proximity­based tokens (e.g., keys and smartcards) for access systems installed at various places such as in cars, at contactless payment terminals, and in electronic passports. Moreover, such systems exist in critical environments like nuclear power plants.Unfortunately, the current systems used to detect proximity between devices and/or users are rife with vulnerabilities. Numerous attacks, such as Relay attack, Preamble Injection attack, Early Detect/Late Commit, and Cicada, exist that let an attacker maliciously alter the measured distance. The research community has proposed several solutions to address these problems and based on their inputs, the IEEE 802.15.4a standard was recently amended. Nevertheless, we show that the newer amendment(i.e., IEEE 802.15.4z) is however not entirely secure and still vulnerable to being exploited.In this work, we evaluate and address the vulnerabilities present in the recently introduced standard,IEEE 802.15.4z amendment for Ultra­Wide Band (UWB). This standard forms the basis of proximity detection in a majority of new devices such as keyfobs for cars, access control systems, smartphones like Samsung S21 and Google Pixel 6, and even medical equipment to monitor patients. First, we mount two attacks, namely the Cicada­TF and the Adaptive Injection, against UWB ­based proximity detection systems. Second, we propose a novel approach to detect the presence of these attacks. We create a real ­world testbed using DWM3000 ICs mounted on NRF52840­devkits to launch the attacks and implement our proposed detection approach. We evaluate the efficacy of our approach in three different environments: an indoor residence, a large outdoor passageway, and an office space. These environments were selected to represent the most commonly used places and were based on the802.15.4a channel models document by IEEE. Our experiment results show that the proposed model can detect the presence of attacks with high accuracy (94%) in all three environments. To the best of our knowledge, this is the first research work that presents a way to detect the presence of such attacks and also to be verified on hardware.

Odette was selected as Talent of the Kinderboekenweek 2021, which has the theme Worden Wat Je Wilt. My Talent is Talenwonder, for which a video was created.@en

Network data are essential in applications such as recommender systems, social networks, and sensor networks. A unique characteristic that these data encompass is the coupling between the data values and the underlying network structure on which these data are defined. Graph Neural Networks (GNNs) have been designed as tools to extend the benefits of deep learning approaches to network data. One crucial component of GNNs is the nonlinear component, also known as the activation function. The activation function allows capturing the nonlinear relationships present in the input data. However, in the current literature, the essential data-network topology coupling is ignored in the nonlinear component of the GNN. To address this limitation, we propose in this thesis a new family of activation functions for GNNs that account for the graph structure and capture the data-network topology coupling, while also allowing for a distributed implementation. Specifically, we propose an initial diffusion of the data over the graph, prior to the local nonlinearization operation. The nonlinearization is designed in a form akin to graph convolutions. The latter leads to a graph-adaptive trainable nonlinear component of the GNN that can be implemented directly or via kernel transformations, therefore, enriching the class of functions to represent the network data. Whether in the direct or kernel form, we show the permutation equivariance property is always preserved. This ensures the output of the GNN is independent of node labeling and that the GNN exploits graph symmetries to generalize to different graphs sharing similar symmetries. Numerical experiments with distributed source localization, finite-time consensus, and distributed regression demonstrate the applicability of the proposed graph-adaptive activation functions in distributed scenarios and show improved or comparable performance to pointwise as well as state-of-the-art localized nonlinearities. Our findings also suggest the benefits of the proposed activation functions in situations where the communication resources in the network are limited.

Time-varying network data are essential in several real-world applications, such as temperature forecasting and earthquake classification. Spatial and temporal dependencies characterize these data and, therefore, conventional machine learning tools often fail to learn these joint correlations from data. On the one hand, hybrid models to learn from time-varying network data combine several specialized models able to capture these dependencies separately, thus ignoring their joint spatio-temporal interactions. On the other hand, state-of-the-art approaches for jointly learning time-varying network data do not exploit the useful prior provided by a graph-time product graph. This prior structural knowledge can aid learning and help the models improve their performance. For this reason, we propose a novel neural network architecture to learn from time-varying network data using product graphs and graph convolutions, thus exploiting this prior during learning. In particular, our architecture (i) learns the most suitable graph-time product graph to represent the time-varying network data and model the graph-time interactions; (ii) performs graph convolutions over this product graph to learn the graph-time interactions from data; (iii) employs graph-time pooling to reduce the dimensionality over layers. To the best of our knowledge, no research has yet attempted to capture spatial and temporal dependencies using graph convolutions over product graphs. Results on synthetic and real-world data show that the proposed method is useful in learning from time-varying network data for both regression and classification tasks. For classification, we cure a real-world dataset for earthquake classification and compare the proposed approach with state-of-the-art models. Results indicate that graph-aware models outperform graph-unaware models on this task. For regression, we evaluate the proposed method on two real-world temperature datasets and compare our architecture with graph-aware and graph-unaware models. Results on the smaller dataset show that linear models outperform neural-network models. On the larger dataset, we find that prior knowledge about graph-time interactions seems to be less beneficial in case of abundance of data. For the synthetic experiments, we find that (i) learning the structure of the graph-time product graph from data improves the performance compared to adopting a fixed type of product graph; (ii) learning sparser graph-time product graphs further improves the performance; (iii) the proposed graph-time pooling technique contributes to the model's generalization capabilities.