Sample selection bias is a well-known problem in machine learning, where the source and target data distributions differ, leading to biased predictions and difficulties in generalization. This bias presents significant challenges for modern machine learning algorithms. To tackle this problem, researchers have focused on developing domain adaptation techniques that aim to create robust methods against sample selection bias. One approach is the use of minimax estimation techniques, which belong to the category of inference-based techniques. Despite the extensive research in developing these domain adaptation methods, there remains a critical need to evaluate their performance. This thesis explores the performance differences of various minimax estimation techniques in the presence of sample selection bias, providing insights into their effectiveness in mitigating the challenges posed by biased data. By understanding and evaluating the performance of these techniques, this research contributes to the advancement of domain adaptation methods and their application in real-world machine learning scenarios.

This paper tackles the problem of sample selection bias in machine learning, where the assumption of train and test sets being drawn from the same distribution is often violated. Existing solutions in domain adaptation, such as semi-supervised learning techniques, aim to correct this bias, but their ability to generalize to unseen test sets remains unexplored. To address this issue, specific semi-supervised methods (self-training and co-training) are trained on biased training sets and tested with an unbiased test set drawn from the same distribution. The results of this paper demonstrate that the semi-supervised methods consistently outperformed or matched the baseline models, with self-training exhibiting greater improvement. Through this study, a promising approach is presented to mitigate sample selection bias in machine learning.

Importance weighting is a class of domain adaptation techniques for machine learning, which aims to correct the discrepancy in distribution between the train and test datasets, often caused by sample selection bias. In doing so, it frequently uses unlabeled data from the test set. However, this approach has certain drawbacks: it requires retraining for each new test set and fails when the number of test samples is very small. Therefore, we seek to study the performance of importance weighting techniques when the unlabeled data comes from an underlying domain, instead of one specific test set. We propose an evaluation framework inspired from scenarios traditionally known for posing difficulties to importance weighting and apply it to two popular algorithms, KMM and KLIEP. Our results reveal that both algorithms produce statistically significant classification improvements in most experiments. However, their performance is highly dependent on the characteristics of the dataset and the sampling bias. In particular, class overlap seems to influence adaptation ability in the case of unequal conditional probabilities of the source and target domains, while the "intensity" of the sampling bias is an important confounding factor when the train set size is small.

Domain adaptation allows machine learning models to perform well in a domain that is different from the available train data. This non-trivial task is approached in many ways and often relies on assumptions about the source (train) and target (test) domains. Unsupervised domain adaptation uses unlabeled target data to mitigate a shift or bias difference between the domains. Deep domain adaptation (DDA) is a powerful class of these methods, which utilizes deep learning to extract high-level features that are common across the domains and robust against the shift. These algorithms adapt to a specific target domain. This has two possible downsides. Firstly, the model might not generalize and thus require retraining for each new domain. Secondly, obtaining data for the target domain(s) might be difficult. There can be situations where both source and target domains originate from a “global” domain, from which the samples are selected in a biased way. We explore a new application of existing DDA methods and answer the question: How effective is deep domain adaptation when adapting with the global domain, instead of the target domain, in case of sample selection bias?. Results with synthetic data show that where target adaptation works, global adaptation also improves accuracy compared to supervised learning, although to a lesser extent.

Sample selection bias occurs when the selected samples in a subset of the original data set follow a different distribution than the samples from the original data set. This type of bias in the training set could result in a classifier being unable to predict samples from a testing data set optimally. Domain adaptation techniques try to adapt classifiers to a possible bias in the training or testing set. Subspace mapping techniques specifically do this by trying to find common subspaces between the source and target domain, where the source domain is the domain with all samples used for training, and the target domain is the domain with samples that must be predicted. This project aims to evaluate the effectiveness of two subspace mapping techniques in mitigating sample selection bias. This research assumes that no data samples from a target domain are available, but only unlabelled samples coming from an underlying global domain. The two subspace mapping techniques that will be tested in this paper are subspace alignment (SA) and transfer component analysis (TCA). This paper will show that the subspace alignment method is more effective on data sets with fewer features and where the source and target domains are further away from each other. The transfer component analysis method is more effective when more training samples are available on data sets with fewer features and where the distance between the source and target domain is not too big. The effectiveness of both methods also depends on the type and form of the data sets they are used on.