Neural networks are commonly initialized to keep the theoretical variance of the hidden pre-activations constant, in order to avoid the vanishing and exploding gradient problem. Though this condition is necessary to train very deep networks, numerous analyses showed that it is not sufficient. We explain this fact by analyzing the behavior of the empirical variance which is more meaningful in practice of data sets of finite size. We demonstrate its discrepancy with the theoretical variance which grows with depth. We study the output distribution of neural networks at initialization in terms of its kurtosis which we find to grow to infinity with increasing depth even if the theoretical variance stays constant. The result of this is that the empirical variance vanishes: its asymptotic distribution converges in probability to zero. Our analysis, which studies increased dependence of outputs, focuses on fully-connected ReLU networks with He initialization, but we hypothesize that many more random weight initialization methods suffer from either vanishing or exploding empirical variance. We support this hypothesis experimentally and demonstrate the failure of state-of-the-art random initialization methods in very deep regimes.

In today's society, claims are everywhere, in the online and offline world. Fact-checking models can check these claims and predict if a claim is true or false, but how can these models be checked? Post-hoc XAI feature attribution methods can be used for this. These methods give scores indicating the influence of the individual tokens on the model's decision-making. In our research, we evaluate three popular feature attribution methods in the context of fact-checking: LIME, Kernel SHAP, and Integrated Gradients. We used the NLP architecture ExPred as a fact-checking model in our research. The feature attribution methods were evaluated using a human-grounded and pseudo ground truth evaluation. The results from these evaluations indicate that Integrated Gradients enables humans to form an opinion better and performs better in our pseudo ground truth evaluation. A potential explanation is that the iterations should be increased for LIME and Kernel SHAP. Our findings suggest that Integrated Gradients performs better in our study. Still, more research for other tasks and models would be beneficial to ensure that these results apply to other cases.

The goal of this paper is to examine how different presentation strategies of Explanainable Artificial Intelligence (XAI) explanation methods for textual data affect non-expert understanding in the context of fact-checking. The importance of understand- ing the decision of an Artificial Intelligence (AI) in human-AI interaction and the need for effective explanation methods to improve trust in AI models are highlighted. The study focuses on three expla- nation methods: interpretable-by-design model Ex- Pred, feature attribution technique LIME, and in- stance attribution method k-NN. Two presentation strategies were compared for each method, and par- ticipants were presented with a set of claims and asked to indicate their understanding and level of agreement with the AI’s classification. The main hypothesis is that participants will appreciate all available context and details, as long as it is pre- sented in a structured way, and will find visual rep- resentations of data easier to understand than tex- tual ones. Results from the study indicate that par- ticipants prefer explanations that are simple and structured, and that visual presentations are not as effective, especially when it is the first time a user interacts with this type of data. Additionally, it was found that better formatting leads to a better- calibrated understanding of the explanation. The results of this study will provide valuable insight into the best way to present XAI explanations to non-experts to enhance their understanding and re- duce the deployment risk associated with Natural Language Processing (NLP) models for automated fact-checking. The study’s code, data, and Figma templates are publicly available for reproducibility.

Deep-learning (DL) models could greatly advance the automation of fact-checking, yet have not widely been adopted by the public because of their hard-to-explain nature. Although various techniques have been proposed to use local explanations for the behaviour of DL models, little attention has been paid to global explanations.
In response, we investigate whether a frequent sequence mining (FSM) tool finds sequence patterns, that act as shortcuts, to a state-of-the-art model in the context of fact-checking. By studying the connections between a model’s input and output, association rules (ARs) can be used as a global explanation for the interpretation of the model. The shortcuts were evaluated using a heuristic-based minimum support value, the strength of each rule was determined using confidence, and the support value indicates the global coverage of rules. Shortcuts help to form an interpretation for creating counterfactual prompts, which can be used as a risk assessment tool for DL models. Other applications for rule-based global explanations are left for future work

Recently, there has been an increase in literature about the Double Descent phenomenon for heavily over-parameterized models. Double Descent refers to the shape of the test risk curve, which can show a second descent in the over-parameterized regime, resulting in the remarkable combination of both low training and low test risk. However, much is still unknown about this behaviour. In this thesis we consider Double Descent and more specifically 'beneficial overfitting', meaning that the lowest test risk as a function of the number of parameters is achieved in the over-parameterized regime. We are mainly interested in under what conditions beneficial overfitting occurs. We start by exploring the test risk behaviour for simple linear regression models, with isotropic Gaussian, general Gaussian and sub-Gaussian covariates. For random feature selection and isotropic covariance, beneficial overfitting occurs for a large signal-to-noise ratio. For deterministic feature selection and isotropic covariance, beneficial overfitting occurs if we select features corresponding to the lowest weights. Without feature selection, beneficial overfitting occurs if the eigenvalues of the covariance matrix have a long, flat tail. In the second part of this thesis we check whether the same or similar results can be applied to other models as well. Specifically, we look at kernel regression, random Fourier features and a classification model. It seems that the linear regression results agree with the random Fourier features model, linear and quadratic kernel regression and classification model, but are not applicable for the Gaussian kernel regression case. Hence, more factors need to be considered, besides the eigenvalue behaviour of the covariance or kernel matrix and the way in which features are selected, to fully explain Double Descent and beneficial overfitting.

Machine learning is becoming an increasingly important tool for climate scientists, but hampered by lacking uncertainty quantification. Here, a machine learning approach for detecting patterns indicating a changing climate is combined with probabilistic modelling to retrieve uncertainty values. We train neural networks on climate model simulations of temperature and precipitation under historical and future scenarios. We find that the resulting so-called Bayesian neural network (BNN) has similar predictive strength to an Artificial neural network (ANN), with a post-year 2000 mean absolute error of 9.00 years for temperature, but over-fits less. The BNN is able to recognise temperature change starting in 1994, which is 14 years later than the ANN. Our analysis shows that uncertainties in found climate patterns are much higher than the patterns themselves, reducing their value for further use. This work demonstrates that BNNs are a suitable tool for quantifying uncertainties of patterns indicating a changing climate.

Neural networks (NNs) have, in recent years, become a major part of modern pattern recognition, and both theoretical and applied research evolve at an astounding pace. NNs are usually trained via gradient descent (GD), but research has shown that GD is not always capable of training very small networks. As a result, networks trained via GD are often significantly larger than necessary, demanding more computing power and energy to evaluate, and thereby hampering adoption on lower- power devices. This thesis investigates whether evolutionary algorithms (EAs) can successfully train NNs and if these networks can be smaller than those required by GD. Four algorithms, namely the GD-based Adam, Adam with cold restarts, and the EAs GOMEA and BIPOP-CMA-ES, were used to train various configurations of multilayer perceptrons (MLPs) with one hidden layer for the Exclusive-OR (XOR) problem. Their relative performance was gauged by comparing the rates at which each algorithm attained an acceptable loss for a given number of hidden nodes. The main findings are that EAs could find smaller ReLU-activated networks than GD could, while, oppositely, the GD could generally find smaller networks for sigmoid- activated networks. However, GOMEA in particular was able to successfully train XOR networks using the highly discrete Heaviside activation function, whereas GD could not due gradient erasure. Problem-specific knowledge in the form of a soft symmetry breaking constraint was found to be effective to increase success rates in a limited number of cases. These findings indicate that using EAs is a viable strategy for training NNs, yielding smaller, and therefore more efficient networks than those trained via GD, provided that the network topology is suitable for the chosen EA. This opens up future research into, including but not limited to, the many ways EAs can be scaled up to training larger NNs, hybridization with GD, and niche network topologies.

Rhyming words are one of the most important features in poems. They add rhythm to a poem, and poets use this literary device to portray emotion and meaning to their readers. Thus, detecting rhyming words will aid in adding emotions and enhancing readability when generating poems. Previous studies have been done on the topic of poem generation. However, those works did not put too much emphasis on the rhyme detector. Thus, this research will solely focus on rhyme detection and its evaluation. The aim of this research is to determine the most accurate way of detecting whether two English words rhyme. English rhyming words will be detected using combinations of features. Five features are used: edit distance, hamming distance, jaccard similarity, longest common substring, and vowel and consonant weights. We also experiment with two methods of retrieving phonemes: using the entire phoneme translation, and using part of the phoneme translation. We find that using only hamming distance and jaccard similarity with part of the phoneme translation, we can already obtain an accuracy of 90.05% with a log loss of 0.25 when trained on a balanced dataset. The reason for this remains unclear because there is no clear separation between the two classes.

Video understanding has received more attention in the past few years due to the availability of several large-scale video datasets and improvement in the computational power of computers. However, annotating large-scale video datasets are cost-intensive due to their complexity. In this work, we propose a time-efficient video annotation method using spatio-temporal feature similarity and t-SNE dimensionality reduction to make the annotation process more efficient. Placing the same actions from different videos near each other in the two-dimensional space based on feature similarity helps the oracle to group label the video clips. We evaluate the performance of our method on two subsets of the ActivityNet (v1.3) dataset. We show that our method can outperform conventional video labeling tools time-wise while maintaining a reasonable test accuracy on video classification task compared to the ground-truth labels. To further evaluate the generalization of our method, we test our performance on Sports-1M and Breakfast datasets.

This thesis presents a novel self-supervised approach of learning visual representations from videos containing human actions. Our approach tackles the complex problem of learning without the need of labeled data by exploring to what extent the ideas successfully used for images can be transferred, adapted and extended to videos for action recognition purposes. We begin by giving a brief introduction to the topic of learning features without having access to a labeled corpora, providing the motivation of our work. We continue with presenting the related research in terms of contrastive learning, action recognition from videos with 3D convolutions and self-supervised techniques for both images and videos. Next, we formalize our approach with regards to the sampling method, the types of spatial and temporal transformations and the contrastive loss used. We evaluate videoSimCLR proposed method in terms of linear evaluation, fully fine-tuning and video retrieval using two popular action recognition datasets, HMDB51 and UCF101. We also explore the extension of another contrastive learning approach to videos, videoMOCO, and compare it with videoSimCLR by means of linear evaluation.

In the present day we use machine learning for sensitive tasks that require models to be both understandable and robust. Although traditional models such as decision trees are understandable, they suffer from adversarial attacks. When a decision tree is used to differentiate between a user's benign and malicious behavior, an adversarial attack allows the user to effectively evade the model by perturbing the inputs the model receives. We can use algorithms that take adversarial attacks into account to fit trees that are more robust. In this work we propose an algorithm that is two orders of magnitudes faster and scores 4.3% better on accuracy against adversaries moving all samples than the state-of-the-art work while accepting an intuitive and permissible threat model. Where previous threat models were limited to distance norms, we allow each feature to be perturbed with a user-specified threat model specifying either a maximum distance or constraints on the direction of perturbation. Additionally we introduce two hyperparameters rho and phi that can control the trade-off between accuracy vs robustness and accuracy vs fairness respectively. Using the hyperparameters we can train models with less than 5% difference in false positive rate between population groups while scoring on average 2.4% higher on accuracy against adversarial attacks. Lastly, we show that our decision trees perform similarly to more complex random forests of fair and robust decision trees.

Many applications employ models to represent real-life environments efficiently. To allow these models to be realistic it is commonly fitted using a dataset containing labeled samples. When obtaining a label for a sample from the environment is expensive, it is key that the dataset contains only those samples that aid in providing a realistic model the most. Active Learning (AL) provides searching strategies for selecting these samples based on different heuristics: diversity, informativeness, and representativeness. This thesis focuses specifically on population-based AL for regression, where both sample and output space are infinite. Its goal is to create a performant, efficient, extensible, and generally applicable selection strategy for this setting. To allow for the latter a black-box model, through which its strategy can be used with virtually any model. The strategy itself is modular, allowing for extensions. This strategy iteratively concentrates on an interesting subregion within the sampling space through three modular steps: discretizing the sample space, providing fitness scores to this discretization, and restricting the sample space based on these fitness scores further. This strategy is applied to both a scientific polynomial setting, as well as a car-following setting. Experiments show that this approach outperforms randomly selecting a sample in both cases, especially when a long labeling time is considered.

Rainfall is increasing in frequency and intensity due to climate change. Hydrological models exist that can report bottlenecks in urban infrastructures. However, these require accurate rainfall estimations with high temporal and spatial resolution. The fulfillment of these requirements is challenged due to high costs. This can be solved with cost-effective acoustic rainfall sensors, that can be densely distributed over a city. A (machine learning) prediction model is then needed to infer rainfall from acoustic data. It is infeasible to create or calibrate a seperate model to the specific characteristics of every acoustic sensor. This research focuses on domain adaptation techniques that transfer a model designed for one sensor, called the source, to a characteristically different one, called the target. A two-stage regressor model is devised that estimates rainfall intensity from acoustic data. In a supervised learning setting, this gave an average $R^2$ score of 0.67. Transfer Component Analysis (TCA) has been used as a semi-supervised domain adaptation technique, where no annotations from the target sensor are needed. This resulted in an average $R^2$ of 0.59. Finally, a supervised domain adaptation technique, where some annotations are used, is proposed using TCA and a mean matching method. Two flavours are researched. In the batched approach a small selection of target annotations are gathered, where after the rainfall is predicted with a delay. The online approach predicts in real-time, but adjusts the prediction model when a new target annotation is available. These resulted in an average $R^2$ of 0.52 and 0.32, respectively.

Deep learning is proving to be a useful tool in solving problems from various domains. Despite a rich research activity leading to numerous interesting deep learning models, recent large scale studies have shown that with hyperparameter optimization it is hard to distinguish these models based on their final performance. Hyperparameter optimization has shown to improve the state of the art results on several occasions. These results cast the doubts over the performance of these improved deep learning models and lead to the question whether the final performance of a deep learning model is dependent on the person performing the hyperparameter optimization task. A user study was conducted to evaluate the impact of human's prior experience in deep learning on the final performance of a deep learning model. 31 people with different levels of experience in deep learning were invited to perform a hyperparameter optimization task. The collected data was analyzed to find the relationship between human and the final performance of the deep learning model used for the user study. From the results, we observed that the final performance of the model vary with every participant, and a strong correlation between the participant's experience and the final performance achieved. Our data suggest that an experienced participant finds better results using fewer resources.

With the huge amount of data that is collected every day and shared on the internet, many recent studies have focused on methods to make multimedia browsing simple and efficient, investigating techniques for automatic multimedia analysis. This work specifically delves into the case of information extraction from videos, which is still an open challenge due to the combination of their semantic complexity and dynamic nature. The majority of the existing solutions are tailored for specific video categories and result in the creation of key frames time-lapses, video summaries, video overviews or highlight clips. In particular, this thesis project focuses on the case of highlights extraction from videos where one person speaks facing the camera. Automating the process of analysis of this specific kind of videos is important in the industrial context because it can be harnessed for several interesting applications, such as the automatic video summarisation of interviews or the automatic creation of personal video curricula vitae.

In this setting, the research objective is to investigate how Machine Learning can be deployed for the task of information extraction. From the target videos, multiple types of features can be extracted, such as textual features from the speech transcription; visual features from the facial expressions, head pose, eye gaze and hand gestures; audio features from the variations in the tone of the voice. ¬The exploitation of multimodal features enhances the capacity of Machine Learning algorithms. In fact, as proven in former research, the integration of multiple channels of information --- textual, audio, visual --- makes it possible to derive a more precise and greater amount of knowledge, just like humans exploit their multiple senses, in addition to experience, to make classifications or predictions. In this work, two approaches for multimodal information extraction from videos are investigated. The first approach is based on simple multimodal feature vectors concatenation, while the second approach exploits a recent deep architecture, the Memory Fusion Network by Zadeh, Amir, et al., to model both individual and combined temporal dynamics. To test the effectiveness of multimodal learning in the context of information extraction from videos, the two techniques are compared against a unimodal, content-based method, that relies on the summarisation of the video transcripts.

In order to train the multimodal approaches in a supervised fashion, a novel dataset based on videos of political speeches of well-known American politicians, the Political Speeches Dataset, was collected. The dataset is provided with binary saliency labels, that allow to identify the ground truth salient video segments. Four types of highlight clips are generated for each speech and evaluated through crowdsourcing. The results show that the quality of automatically created highlight clips is comparable to the ground truth, in terms of informativeness and ability to generate interest. Moreover, they also confirm that highlight clips generated with multimodal learning are more informative than the baseline.

Optical flow is a representation of projected real-world motion of the object between two consecutive images. The optical flow measures the pixel displacement on the image coordinate plane. However, it does not reveal the motion in depth explicitly, which could be useful as input in some tasks such as vehicle tracking. To extend the original optical flow approach, we model the depth change of the object as the scale change of object in the image and present an approach to estimate the scale change and optical flow jointly. Considering the scenario that obvious scale change occurs between two images, the traditional convolution network fails because it lacks the scale invariance. According to the Scale-space theory and the idea of learning a combination of Gaussian derivative basis to approximate arbitrary filters, we build a Basis Convolution layer that allows the network to see the scale change between two images and make use of it to better capture the same feature with various scales on two images.

We test our models on our own optical flow datasets which involve obvious scale change. According to the experimental results, our method is capable of estimating the scale change between images, and it significantly improves the optical flow estimation by modelling the scale change explicitly.

A lot of attention has recently been focused on possible benefits of the cooperation between machines and humans. Taking the best from machines and humans and joining them together can produce results which exceed each collaborating partner performing separately. A common belief is that the
key for good cooperation is an excellent communication. Some important aspects of communication are self-evaluation processes. When applied to humans these processes improve the communication quality between humans. Therefore, we believe that employing self-evaluation processes at machines
advances the human-machine communication, and cooperation quality. Accordingly, this thesis is exploring communication strategies between machines and humans. More precisely, it examines possibilities for the communication improvement through an exploration of self-evaluation processes of classifiers.

Firstly, we introduce a baseline framework, an interactive visual category learning architecture, called Tubby at Honda Research Institute in Germany. For simplicity we consider in a first step the classification of objects rather than the more difficult task of learning multiple categories per object. We then introduce theoretical foundations used in the thesis. The background on classification, neural networks, outlier detection and assessment of classifiers is explained in depth. We outline the critical importance of self-evaluation in classification. Therefore, we propose two self-evaluation measures which are incorporated within a testing and a training strategy. The first measure captures a confidence
in predictions during classification and it is used within the proposed testing strategy. The second measure denotes the quality of each training sample with respect to the generalization performance of classification and it is used within the proposed training strategy. The quality of the training sample essentially represents how different the current training sample is from all of the previously acquired training samples of the object. The confidence and the quality measure are communicated through a graph to the user of the system. Depending on whether the system is in the testing or training phase the value of the corresponding measure is provided to the user. Two ways of deriving each
of the two measures are presented. Essentially, we offer two testing and two training strategies. Furthermore, we consider each proposed strategy separately and a combination of those. We then evaluate all of the considered cases against a baseline strategy. The evaluations are performed in different dimensionalities of the feature space for different numbers of training and testing objects. We present an offline simulation of the interaction between Tubby and the user. Results of the simulation provide additional insights into the working mechanisms of the proposed strategies and measures.

The proposed strategies improve the baseline performance. The absolute improvement of the average classification accuracy varies between 1% and 25%, depending on the dimensionality of the feature space and number of training and testing objects. The best results are achieved when a combination of the proposed training and testing strategy is used. The biggest improvement is observed when a lot of objects are in the learning process (≈ 100), and the dimensionality of the feature space is high (≥ 10D). In the actual application setting this is the most realistic case - a large number of objects and a high dimensionality of the feature space.

Decision-theoretic planning techniques are increasingly being used to obtain (optimal) plans for domains involving uncertainty, which may be present in the form of the controlling agent's actions, its percepts, or exogenous factors in the domain. These techniques build on detailed probabilistic models of the underlying system, for which Markov Decision Processes (MDPs) have become the de facto standard formalism. However, handcrafting these probabilistic models is usually a daunting and error-prone task, requiring expert knowledge on the domain under consideration. Therefore, it is desirable to automate the process of obtaining these models by means of learning algorithms presented with a set of execution traces from the system. Although some work has already been done on crafting such learning algorithms, the state of the art lacks an automated method of configuring their hyperparameters, so to maximize the performance yielded from executing the derived plans.
In this work we present a method that employs the Bayesian Optimization (BO) framework to learn MDPs autonomously from a set of execution traces, optimizing the expected value and performance in simulations over a set of tasks the underlying system is expected to perform. The approach has been tested on learning MDPs for mobile robot navigation, motivated by the significant uncertainty accompanying the robots' actions in this domain.