With the rise of zero-shot synthetic image generation models, such as Stability.ai's Stable Diffusion, OpenAI's DALLE or Google's Imagen, the need for powerful tools to detect synthetic generated images has never been higher. In this thesis we contribute to this goal by considering wavelet-based approaches for synthetic image detection.

We will introduce multi-level discrete wavelet transform, which to the best of our knowledge has never been considered for this goal prior to this work. A similar approach that has been considered for the goal of synthetic image detection, is the multi-level wavelet packet transform used by Wolter et al. We will show that not only is our proposed approach more efficient and easier interpretable, it also performs better in a number of experimental settings and therefore forms a suitable addition to the toolset for the detection of synthetic images.

Moreover, we will try and generalize performance of our used classifiers to out-of-dataset samples and see that our used classifier in general does not allow for such generalization. Finally, we will discuss the challenges of this work and offer interesting directions for further research.

Machine learning algorithms (learners) are typically expected to produce monotone learning curves, meaning that their performance improves as the size of the training dataset increases. However, it is important to note that this behavior is not universally observed. Recently monotonicity of learning curves has gained renewed attention, as several authors have proposed ’wrapper’ algorithms; algorithms that attempt at filtering the hypotheses produced by a learner to turn them into a monotone learner, even if the learner itself is not monotone. Such wrappers use part of the training data as validation data, and each newly produced hypothesis is evaluated using this validation data. However, with each new hypothesis, the validation data grows in size exponentially. As such the wrapper is data-hungry, using up to 85% of the training data as validation data in some cases. This paper investigates what happens when a linearly growing validation sample is used instead. Is it enough to retain monotonicity? We proof that selecting the best performing hypothesis from a finite set of hypotheses, based on a validation sample that grows linearly, results in a monotone learning curve. However, when introducing a new hypothesis with each increase in the validation sample size, it has been observed that this selection process does not demonstrate monotonic behavior. The authors of this paper hope that this work provides key insight into how to choose from a set of hypotheses in a monotone way, and that the work may be a stepping stone for a fully functioning linear-data monotone wrapper algorithm.

A learning curve displays the measure of accuracy/error on test data of a machine learning algorithm trained on different amounts of training data. They can be modeled by parametric curve models that help predict accuracy improvement through curve extrapolation methods. However, these learning curves have only been mainly generated from default learning algorithms. Research into tuning the machine learning algorithm and its effect on the learning curve has not been adequately researched. This research aims to look at the influence of hyperparameter tuning on the learning curve. This regards not only how the learning curve shape changes in general but also how different parametric models are affected when a learner undergoes tuning. We experiment with the decision tree and KNeighbors classifier which undergo significant hyperparameter tuning. We find that the tuned learner performs marginally better than the default learner for anchors past 25\% of the data for the majority of the tested datasets. We also observe that the tuned learner displays a smoothing behaviour that makes ill-behaved curves more well-behaved. In terms of the curve fitting, the tuned learners do not uncover any curve models nor does it show any statistical significance, and instead performs very similarly to the default learners.

Learning curves have been used extensively to analyse learners' behaviour and practical tasks such as model selection, speeding up training and tuning models. Nonetheless, we still have a relatively limited understanding of the behaviour of learning curves themselves, in particular, whether there exists a parametric function that can best model all learning curves. Therefore, this study aims to determine which parametric models proposed over the years provide the best fit when applied to empirical learning curves. To answer this question, the study focuses on supervised learning and is divided into two parts: classification and regression tasks, and the learning curve data for each task was fitted using the Levenberg-Marquardt algorithm. Subsequently, the fitted models were analysed using the Friedman test, the Wilcoxon signed-rank test, and other metrics. The results indicate that a power law applies in most cases. However, a universal model has not been found, as the best model differs between classification and regression tasks, even though they belong to the power law family. Moreover, there are some deviations from these aggregate results when examining the learners individually, suggesting that a more granular approach is better suited for practical applications.

Extrapolation of the learning curve provides an estimation of how much data is needed to achieve the desired performance. It can be beneficial when gathering data is complex, or computation resource is limited. One of the essential processes of learning curve extrapolation is curve fitting. This research first analyses the behaviour of existing curve fitting methods such as Newton, Levenberg-Marquardt and Evolutionary algorithms when fitting different function models on learning curves. Furthermore, it also illustrates a few techniques to improve the learning curve fitting and extrapolation procedure.

The learning curve illustrates how the generalization performance of the learner evolves with more training data. It can predict the amount of data needed for decent accuracy and the highest achievable accuracy. However, the behavior of learning curves is not well understood. Many assume that the more training data provided, the better the learner performs. However, many counter-examples exist for both classical machine learning algorithms and deep neural networks. As presented in previous works, even the learning curves for simple problems using classical machine learning algorithms have unexpected behaviors. In this paper, we will explain what caused the odd learning curves generated while using ERM to solve two regression problems. Loog et al. [1] first proposed these two problems. As a result of our study, we conclude that the unexpected behaviors of the learning curves under these two problem settings are caused by incorrect modeling or the correlation between the expected risk and the output of the learner.

Autoencoders seek to encode their input into a bottleneck of latent neurons, and then decode it to reconstruct the input. However, if the input data has an intrinsic dimension (ID) smaller than the number of latent neurons in the bottleneck, this encoding becomes redundant.
In this paper, we study using the Early-Bird (EB) technique, a structural pruning method, to regularise and prune the redundant latent neurons. We do this for both linear-layer and convolutional autoencoders, on 1D and 2D data. We find that increasing the strength of EB regularisation specifically on the latent layer can lead to all redundant latent neurons (and no more) being removed in one training run.
We also compare using EB in this manner to existing ID estimation methods: we find it performs comparable to older methods like local-PCA, also being relatively robust to noise, but that it does not match the best existing ID estimation methods.

Although there are many promising applications of a learning curve in machine learning, such as model selection, we still know very little about what factors influence their behaviours. The aim is to study the impact of the inherent characteristics of the datasets on the learning shapes, which are noise, discretized input and dimensionality. We trained two classifiers with a panoply of datasets for the investigation to see how the learning curve behaves under different circumstances. Firstly, we found that the shapes of the curves varied with different levels of noise injected into the original datasets. Secondly, using the equal width interval binning technique to discretize continuous features did not make the classifiers learn exponentially but caused the learning curves to behave unpredictably; thus, it does not transform the continuous problem into the easier class of problems mentioned in [1]. Finally, the more dimension we reduced using the PCA technique, the learning curve showed strange behaviours.

Supervised machine learning is a growing assistive framework for professional decision-making. Yet bias that causes unfair discrimination has already been presented in the datasets. This research proposes a method to reduce model unfairness during the machine learning training process without altering the sample value or the prediction value. Using an objective function that identifies the biased feature with maximal correlation estimation, the method selects samples to train the updated classifier model. The quality of the sample selection determines the extent of unfairness reduction. With an adequate sample size, we demonstrate that the method is valid in reducing model unfairness without severely sacrificing classification accuracy. We tested our method on multiple benchmark datasets with demographic parity and feature independence as the notions for a statistically fair classification model.

Does a convolutional neural network (CNN) always have to be deep to learn a task? This is an important question as deeper networks are generally harder to train. We trained shallow and deep CNNs and evaluated their performance on simple regression tasks, such as computing the mean pixel value of an image. For these simple tasks we show that going deeper does not guarantee an improvement in performance.

This research provides an overview on how training Convolutional Neural Networks (CNNs) on imbalanced datasets affect the performance of the CNNs. Datasets could be imbalanced as a result of several reasons. There are for example naturally less samples of rare diseases. Since the network is trained less on those instances, this might lead to worse performance on those cases. However, it might be more crucial to identify those cases properly. Furthermore, it is non-trivial to check whether real-time generated data is balanced. The networks in this research are trained on three different types of synthetic datasets. Balanced datasets, datasets with missing targets and datasets that have normally distributed targets. The task of the network is to find the standard deviation of the pixel intensity of the input. The results show that it is best to train the network on balanced datasets, however training networks on datasets with normally distributed targets does not result in a big loss. Furthermore, in this case the CNNs were still able to learn the task with decent performance if the training set missed targets.

With an expectation of 8.3 trillion photos stored in 2021 [1], convolutional neural networks (CNN) are beginning to be preeminent in the field of image recognition. However, with this deep neural network (DNN) still being seen as a black box, it is hard to fully employ its capabilities. A need to tune hyperparameters is required to have a robust CNN that can more accurately do its task. In this study, the batch size, being one of the most important hyperparameters, is our main concern. The batch size is the number of samples that will be propagated through the network before updating the weights. Moreover, we show how the batch affects the performance of Regression CNNs to the following regression tasks: the mean, median, standard deviation (std) and variance of the pixel intensities of a grey-scale MNIST [2] input image. This will be analyzed by how well regression CNNs converge, given different batch sizes and a fixed learning rate. Additionally, we will also be comparing the final mean squared error given by all different batch sizes. At the end of the research, our findings concluded that a higher batch size leads to a higher Mean Squared Error (MSE) and a slower convergence. Additionally, the best performance obtained was for batch sizes of size 8 to 32, with slight differences between the four different regressions tasks.

Yes, convolutional neural networks are domain-invariant, albeit to some limited extent. We explored the performance impact of domain shift for convolutional neural networks. We did this by designing new synthetic tasks, for which the network’s task was to map images to their mean, median, standard deviation, and variance pixel intensities. We find that the performance drop due to domain shift is related to the shift in pixel values between source and target domain. Colour space transformations seemed to notably impact the network’s performance, opposed to geometric transformations. For the last domain shift we find that the network manages to beat a baseline, from which we can conclude the domain shift is not too severe. Additionally, the findings reveal a less dominant role for feature transferability, for our synthetic regression tasks.

This paper compares the performance of two phonetic notations, IPA and ASJPcode, with the alphabetical notation for word-level language identification. Two machine learning models, a Multilayer Percerptron and a Logistic Regression model, are used to classify words using each of the three notations. With both models the IPA notation outperforms the other two notations.

Currently, trained machine learning models are readily available, but their training data might not be (for example due to privacy reasons). This thesis investigates how pre-trained models can be combined for performance on all their source domains, without access to data. This problem is formulated as a Multiple-Source Domain Adaptation (MSA) problem setting, where models trained on source domains are combined so that the combiner is robust to application on any unknown target domain. This thesis explores the MSA setting and presents a perspective on MSA theory from literature. The issue in the MSA setting is that target models are not robust in general, leading to negative transfer. Firstly, this issue is illustrated by example of the source models and linearly weighted combinations of the source models. Next, existing theory that guarantees the existence of a robust model is investigated. It is argued that a performance bound has the potential to be extended from the perspective of additional knowledge: in addition to the in the MSA setting available source models, some additional knowledge of the source domains might be used by the model. Existing MSA theory’s assumptions are clarified and the theory is split in two. One half is inherent to the MSA setting and guarantees a model with as robustness property the loss on a matching mixture. The other half assumes a combiner that uses additional knowledge of the source domains, for which the robustness is proven satisfactory. Finally, it is investigated what makes additional knowledge in the MSA setting useful. Current literature assumes a specific target model--the distribution-weighted (DW) combiner--that is viewed as using the sources' joint distributions as additional knowledge. It is argued that knowledge of the training process of the source models can also be used as additional knowledge. In conclusion, this thesis discusses how robustness in the MSA setting can be improved from that of the source models by basing the combiner on additional knowledge of the source domains.

Active learning has the potential to reduce labeling costs in terms of time and money. In practical use, active learning works as an efficient data labeling strategy. Another point of view to look at active learning is to consider active learning as a learning problem, where the training data is queried by the active learner. Under this perspective, an important question is inconsistency: can classifiers trained using active learning converge to the same result as using random sampling given an infinite number of data. In this paper, we discuss the possibility and potential consequences of using new sampling settings other than sampling without replacement in active learning to analyze the inconsistency problem. Moreover, a third sampling setting is defined to simulate the infinite data scenario in inconsistency. We compare the traditional setting, sampling without replacement in active learning with sampling with replacement in active learning, and true active learning. Furthermore, the two unusual sampling settings provide insight into the inconsistency problem. (1)Regularization parameter without adjustment can lead to inconsistency. (2)Querying data ”really” close to the decision boundary can also bring threats to active learning.

An increasing number of photovoltaic (PV) systems are being installed worldwide and residential sector is responsible for a large part of this growth. Small scale PV systems do not have complex measuring devices and their breakdowns are not spotted immediately by the system owners. This might lead to prolonged time without generating power and creating both financial loss and environmental damage. This thesis presents a method of PV yield nowcasting laying foundations for remote monitoring. Early detection of faults is the first step towards eliminating the described issues. In this project four machine learning models for predicting solar yield were developed: ElasticNet, Polynomial Regression, Random Forest, and Extreme Gradient Boosting (XGBoost). The models were created both for daily and hourly data sets, as some inverters can log daily yields only. In both cases, the utilized data set consisted of data for the time between July 1st, 2018 and June 30th, 2019 and corresponding to 1,102 PV systems which is five times more than the largest data set studied in the found literature. The average PV system size in the data set is 4.44 kWp. Utilized inputs next to weather data and previous yields included shading factor describing fraction of direct light unable to reach PV system due to surrounding obstacles. Calculation of shading facor was based on 360⁰ pictures taken at the site. XGBoost algorithm turned out to be the most suitable for the task of PV yield nowcasting obtaining RMSE of 1.48 kWh and MAE of 0.877 kWh for hourly data aggregated to daily values and evaluated on future time steps. Currently used commercial software of Solar Monkey has RMSE equal 2.237 kWh and MAE equal 1.5 kWh. XGBoost model trained on daily data obtained RMSE 1.185 kWh and MAE 0.698 kWh outperforming hourly model most likely due to utilization of Hidden Markov Model for data cleaning. Next to overall performance, per system metrics were calculated for the hourly XGBoost. Mean individual RMSE for previously seen systems is 1.656 kWh while for unseen systems it equals 1.666 kWh. This means the model scales well to previously unseen systems and implies that its parallelized version is not necessary. Also, the model's learning saturates after seeing data corresponding to one year and 278 PV systems. Precalculation of GPOA worsened performance with respect to the model utilizing GHI. Hourly XGBoost has hourly RMSE of 0.281 kWh under clear sky and 0.377 kWh under partly cloudy sky which indicates it is more mistaken for cloudy conditions. This could be caused by low quality of cloud coverage data. The model also has large relative errors for small irradiance values which occur mostly in January and December, as well as just after sunrise and just before sunset. This issue is caused by using squared error as loss function during model training. Despite these shortcomings, the conclusive results recommend industrial implementation of the developed model.

The core challenge of the BedBasedEcho BEP project is to create an algorithm to find the heart, and apply it on a robotic echocardiography solution. The team has found multiple complex solutions that are related to this problem, and has extracted useful information from these solutions to apply to this problem. However, some of these complex solutions were too complex, causing the team to run out of physical resources, or to have the solution fail entirely. By taking a step back, and simplifying the solution, the team has managed to create a system that performs marginally better than the complex solutions. The designed product consists of three major components: the data gathering, the learning, and the deployment. When used in this order, the result is an algorithm that can predict which way it should move to gain the optimal view of the heart. The algorithm will be used as a component in a larger automated echocardiography system. Ultimately, the algorithm showed promise by autonomously finding a good view of the heart.

Records from ledgers of Dutch companies all across the Netherlands are used in this study. Records can be submitted in the ledgers with various lags, because the data of many different bookkeepers is involved with different workflows. Bookkeepers can be punctual or late, therefore records can be submitted with various lags in the ledgers. This causes missing data, which results in a deformation of a time series that is constructed from these records. Using a technique called now­casting, a prediction can be made of how these series with no missing data would have looked like.
This study sheds light on how information of an incomplete time series from ledgers of Dutch companies can be used to now­cast on that series, without the use of external indicators. To better utilize the information available from the series, an addition to the Seasonal Auto Regressive Integrated Moving Average with eXogenous regressors (SARIMAX) model is proposed. The addition to the SARIMAX model is presented in two forms: the additive­ and multiplicative relation between indicator­ and target series. These are modeled with the goal to improve the information utilization and therefore improve the now­casting accuracy. Experiments have shown that this addition to the model does not give a direct improvement in accuracy compared to an ordinary SARIMAX model. Thereafter an iterative now­cast procedure is proposed to utilize information from highly lagged records. It has been shown that this gives a slight increase in accuracy for the overall now­cast.

In recent years many new text generation models have been developed while evaluation of text generation remains a considerable challenge.  Currently, the only metric that is able to fully capture the quality of a generated text is human evaluation, which is expensive and time consuming. One of the most used intrinsic evaluation metrics is perplexity. This paper researched the correspondence between perplexity scores and human evaluation of scripts for the TV-show \textit{Friends} generated using OpenAI's GPT-2 model.  This was done by conducting a survey taken by 226 participants that evaluated selected scripts on creativity, realism and coherence.  The survey results revealed that generations with a perplexity value close to that of an actual Friends script perform best on creativity, but score low on realism and coherence.  The most realistic and coherent generations were those with a lower perplexity value, while the worst in all fields were the generations with the highest perplexity value.  The research shows that perplexity is not an adequate measure for the quality of generated TV-show scripts.