Cyber-physical systems are complex systems constructed from different independent parts. A self-driving car is an example of a cyber-physical system where independent parts have to come together in order to result in a car that is able to drive by itself. The main challenge is finding failures within the interactions between the independent parts of the self-driving system. In this paper, we present a novel algorithm REWOSA, in order to detect these faults within the self-driving software. With the use of real-world roads extracted from Google Maps combined with a multi-objective genetic algorithm, we develop a new way to generate roads for testing self-driving cars. We evaluate this algorithm against a state-of-the-art multi-objective genetic algorithm using randomly generated roads using two different setups. Our results show that REWOSA is able to generate more failures than the baseline on both the setups, as well as create more complex roads. In return, REWOSA does create a large overhead due to the complexity of the real-world roads. However, this overhead is justifiable as we can detect more faults with the more complex real-world roads.

The perpetual desire for more qualitative software has been an excellent incentive for software engineers to create automated tools to ease and improve the process of software testing. EvoSuite is an example of a state-of-the-art tool that synthesises test cases automatically. It uses a genetic algorithm to produce test cases based on given search targets. Previous studies have analysed the performance of single or combinations of targets but have not yet explored the differences between various combinations. In this research, we compare the Weak Mutation + Branch setting to Branch and the Default (combination of eight separate targets) of EvoSuite. We aim to provide insightful information about their differences in branch coverage and mutation scores. Moreover, we discuss machine-learning models that can predict which combination has the highest score (i.e., branch coverage, mutation score) based on characteristics of the tested classes, such as the number of lines of code. Our results highlight that the Weak Mutation + Branch combination outperforms Branch for the mutation score metric and Default for the branch coverage metric. They also show that Weak Mutation + Branch is outperformed by the branch criterion for Branch Coverage and by the Default combination for mutation score. Our findings also cover the performance of the models, having concluded that the Random Forest and Decision Tree classifiers produce the best results and are feasible options for predicting the best combinations from the ones analysed. Finally, static code metrics such as 'wmc', 'loc', and 'mathOperationsOty' often appear as relevant features for our models. We visualise how they influence the most suitable combination of criteria through our Decision Trees.

To ensure that a software system operates in the correct way, it is crucial to test it extensively. Manual software testing is severely time-consuming, and developers often underestimate its importance. Consequently, many tools for automatic test generation have been developed during the past decade. EvoSuite is a state-of-the-art tool for automatic generation of unit tests. It can produce test suites based on chosen coverage criteria, also known as a fitness function. Previous studies have widely assessed the performance of the different fitness functions available in EvoSuite. However, the combination of various coverage criteria has not been considered. In this paper, we assess the effectiveness of the combination of Branch coverage and Output diversity fitness functions. We compare it to two of the most popular fitness functions in EvoSuite - Branch coverage and the Default configuration (combines eight coverage criteria) to estimate its performance. We developed a machine learning tool that determines which fitness function will achieve better results based on class characteristics. The assessment criteria we consider are branch coverage and fault detection, represented by mutation score. We further examined how the time limit affects the performance of the considered fitness functions. The results have shown that the combination of Branch coverage and Output diversity outperforms the Default configuration significantly in branch coverage but has worse performance in fault detection capabilities. We have also found that the Branch and Output diversity coverage criteria achieve better results when compared with only using Branch coverage in terms of mutation score. Additionally, the static software metrics, especially CBO, LCOM* and LOC, are highly correlated with the performance of the fitness functions and can predict which coverage criteria will achieve better results.

Recently, automating test suite generation is a problem that has drown attention in both industry and academia. One of the tools used to automatically generate test suites is EvoSuite, which is a state-of-the-art tool often used in research. It uses a genetic algorithm, which seeks to maximize certain coverage criteria, such as Branch Coverage or Exception Coverage. Previous research has investigated the possibility of combining multiple coverage criteria, but there is no single combination which performs best for all classes. This paper aims to investigate if it possible to predict when coverage criteria perform best, specifically Exception Coverage, according to the characteristics of the class under test. The paper shows that there is a significant difference between the performances of these coverage criteria. The paper also shows that this difference can be predicted using a Machine Learning model with an f1-score of 0,865, when performance was measured using Mutation Score. Lastly, an exploration is made into the characteristics of the Class-under-test which cause this.

Software testing is essential for a successful development process, however, it can be troublesome as manually writing tests can be time demanding and error-prone. EvoSuite is a test case generating tool developed to address this [18]. It can generate test cases for different test criteria - Line Coverage, Branch Coverage, Input Diversity, etc. Branch Coverage puts the focus on covering branches in the code, whilst Input Diversity puts the focus on the use of diverse inputs as parameters in the test cases. The downside is that the user needs to select the best suited test criteria, out of the many that EvoSuite provides, for the classes under test. It is not feasible for the user to manually find the optimal one for the classes under test. This paper aims to shine a light on the effectiveness of the combination of Input Diversity and Branch Coverage as a test criteria. This study presents a machine learning technique to automatically select the best combination of test generation objectives according to static metrics. The model we chose for this task is a decision tree as it directly provides a pattern. Said pattern is a combination of conditions that the static metrics need to hold for the chosen test criteria to be effective. The evaluation of the effectiveness was done one a benchmark of 346 classes taken from SF-110 Corpus of Classes [9] and the Appache Commons. To evaluate the effectiveness of Input Diversity in combination with Branch Coverage, we compared the test criteria to two other test criteria - Branch Coverage and the Default coverage criteria used in EvoSuite. The decision tree models created achieve an accuracy upwards of 90% in the best case and deem metrics such as wmc, dit, fanin and others to be crucial for the effectiveness of Input Diversity in combination with Branch Coverage.

Various search-based test generation techniques have been proposed to automate the process of test generation to fulfill different criteria (e.g., line coverage, branch coverage, mutation score, etc.). Despite these techniques' undeniable accomplishments, they still suffer from a lack of guidance coming from the data gathered from the production phase, which makes the generation of complex test cases harder for the search process. Hence, previous studies introduced many strategies (such as dynamic symbolic execution or seeding) to address this issue. However, the test cases created by these techniques cannot assure the full coverage of the execution paths in software under test. Therefore, this thesis introduces common and uncommon behavior test generation (CUBTG) for search-based unit test generation. CUBTG uses the concept of commonality score, which is a measure of how close an execution path of a generated test case is from reproducing the same common and uncommon execution patterns observed during the real-world usage of the software. To evaluate the performance of CUBTG, we implemented it in EvoSuite and evaluated it on 150 classes from JabRef, an open-source application for managing bibliography references. We found that CUBTG managed to cover more common behaviors than plain MOSA in 75% of the cases, and more uncommon behaviors in 60% of the cases. In up to 10% of the cases CUBTG managed to find more mutants seeded by PIT by using method sequences that plain MOSA did not find.

Software applications inevitably crash, and it is time-consuming to recreate the crash conditions for debugging. Recently, researchers have developed frameworks relying on genetic algorithms, e.g. Botsing, for automated crash reproduction. However, the existing approaches process exceptions of different types as if they were the same. In this thesis, we study how the four most common types of Java exceptions are thrown and define specialised fitness functions for them. We have extended Botsing and carried out an evaluation against 52 real-world crashes from seven various open-source software applications. Our results show that our proposed fitness functions influence both the effectiveness and efficiency, negatively or positively depending on the type of the target exception. This thesis demonstrates how tailoring the fitness functions according to the exception type can improve search-based crash reproduction.