A substantial number of passengers in Europe rely on trains for transportation, facilitated by a network of high-speed international trains. However, the coordination of train schedules across multiple networks often poses challenges due to incompatible timings. The scheduling of multiple train networks shares similarities with multi-processor task scheduling and airline scheduling but is distinguished by its cooperative nature rather than a competitive one. Cooperative scheduling necessitates the sharing of private information. This information, 'demand', is commercial sensitive information, since it can reveal demographic information like incomes and tax returns. Privacy-preserving protocols can enable the computation of statistics without revealing this demands (in railway systems) to unauthorized parties. Despite the critical role of privacy in multi-party scheduling, research in this domain remains limited due to domain specific constraints. A model supporting such privacy considerations could significantly help Europe achieve its carbon-neutral goals while improving cross-border services. In this research, we propose a system designed to facilitate joint service scheduling, ensuring confidentiality, integrity, and authenticity. We use partial and fully homomorphic encryption techniques that mimic the outcomes achievable with a trusted third party. We conduct a comparative analysis of online and offline approaches, emphasizing how they achieve confidentiality, collusion-resistance, traceability and non-repudiation. Theoretical and experimental evaluations demonstrate the feasibility of the system for real-world applications by creating schedules for upto four parties. Our solution for scheduling seven slots takes approximately three hours, which is a feasible duration to solve a problem of this size.

Manually crafting test suites is time-consuming and susceptible to bugs. The automation of this process has the potential to make this task more appealing. While current tools like EvoSuite manage to obtain high coverages, their generated tests are not always readable. Recent literature indicates that Large Language Models (LLMs) could address readability and comprehension issues. Our objective in this study is to explore the capabilities of ChatGPT-3.5-Turbo in enhancing existing Java unit tests. We have designed an algorithm that sends multiple prompts to the LLM and overwrites the test cases with the ones received from GPT-3.5. Thus, we have assessed its performance by measuring the initial mutation score of the test suite with the new coverage. The benchmark consists of 16 non-trivial Java classes, on which we performed 80 runs of our algorithm. The results indicate that after one run, GPT-3.5 increases mutation coverage by 23% on average for isolated classes. However, for classes with dependencies, it is less reliable, often producing code with run-time or compile-time errors. Through this paper, we hope to emphasize the importance of ongoing research in this domain to optimize LLMs for providing better test cases.

Mutation testing is a way to test the effectiveness of a test suite for catching bugs in a given piece of code. Writing these tests manually can be cumbersome and time-consuming. Automated tools can be used to generate tests that achieve a high mutation score. The output of these tools is often very hard to understand for humans, and therefore rarely used as actual test suites for software programs. Because LLMs have been shown to be able to generate programs that can be more easily understood by humans, we ask if these LLMs can be used for improving or generating tests for the purpose of mutation testing. Some LLMs run in the cloud, while others run locally. Cloud-based LLMs such as ChatGPT or Copilot are not always an option because of privacy concerns, speed, or regulations, but do not require possession of hardware. Local LLMs do not have the privacy concerns, but sometimes require large amounts of hardware to be available. This paper will focus on local LLMs that can be run in a computationally restricted environment. We present an automated approach to use a local LLM to improve the mutation score of existing test suites. We compare three different models (DeepSeek Coder, Code Llama and Codestral), evaluated on publicly available datasets. Using this approach, we were able to successfully generate unit tests that, combined with the existing manually written tests, are able to increase the mutation score around one third to half of the time depending on the model.

The automated generation of test suites is crucial for enhancing software quality and efficiency. Manually writing tests is time-consuming and accounts for about 15% of project time while tests generated by automated tools like EvoSuite and Pynguin often lack readability and comprehensibility. Recent research suggests that Large Language Models (LLMs) might offer a promising alternative. This paper investigates the effectiveness of Llama3 70B in generating unit test cases for Java and Python projects. We compared Llama3 against EvoSuite and Pynguin by measuring the mutation score of test suites generated for a corpus of 20 Java and 20 Python classes. Our findings reveal that EvoSuite significantly outperforms Llama3 in terms of mutation score, achieving an average mutation score of 81.05% versus Llama3's 66.26%. Conversely, Llama3 surpasses Pynguin, with scores of 51.95% and 42.73% respectively. These results highlight that while Llama3 is not superior to EvoSuite, it shows potential as a viable tool for test generation, especially for dynamically typed languages like Python. Further, empirical observations indicate that Llama3 requires significantly more time to generate tests compared to EvoSuite and Pynguin. This study underscores the need for continued research to optimize LLMs for software testing and improve their efficiency and accuracy.

Over the last few years, Large Language Models have become remarkably popular in research and in daily use with GPT-4o being the most advanced model from OpenAI as of the publishing of this paper. We assessed its performance in unit test generation using mutation testing. 20 Java classes were selected from the SF110 Corpus of classes, and for each 10 different test classes were generated. After we resolved build errors and removed failing assertions, the evaluation using Pitest produced around 71% of mutation coverage on average on the sample dataset. Manually fixing the failing assertions increased the overall mutation score to 75%. Nonetheless, one of the main drawbacks was the need to manually resolve problems that the GPT-4o responses produced, such as code hallucination and incorrect assumptions about the classes under test.

Finding and fixing software faults is a major part of software development and as such any improvement for such tasks is a welcome aid for developers and a worthwhile field for researchers. Like programming in general, debugging and repair need specialized tools to provide the necessary information (like the usage of runtime resources) or assure quality (e.g. with test suites). Only then, developers are able to repair faults without introducing new ones. There are also more sophisticated tools that provide stronger, more automated help to developers: Program coverage summarizes run-time behavior, fault localization helps to narrow down suspicious parts of the code and automated program repair suggests possible patches that lead to a passing test suite. On top of these approaches, large language models show promising capabilities to generate, alter and test source-code, but they have yet to be tested and hardened for their security and quality.
To enable the next generation of state of the art quality-assurance tools, this thesis investigates different techniques and their respective tools to improve their precision and correctness. To this end, we develop procedures to quantify the robustness of large language models of code to identify their weaknesses when facing metamorphic noise or statistically unlikely data. After examining quality of tools, this dissertation works towards improving existing tools and approaches in the field of functional programming, particularly for Haskell. Functional programming is a field rich of unique options such as properties, strong type-systems, side-effect free functions, but also challenges like non-strict evaluation.
Our results regarding large language models show that there are short-comings when dealing with redundant elements and that such elements can be intentionally searched for. This implies a need for further improvement of the models, to provide more consistent results for trivial changes.
The work centered around Haskell shows the value of utilizing compiler- and language-features to enhance existing techniques: Program repair can be performed with a reduced search space due to compiler-suggested elements, stack-traces and program-coverage can be enhanced by introducing an evaluation-trace and fault-localization is aided by types and expression-level granularity. While the implementation is specific, the approaches remain transferable: Any feature that is used from Haskell in this dissertation, is (or can be) implemented for Java.
In summary, this thesis touches on different topics of assuring software quality and their tools by introducing novel information. This thesis lays groundwork to improve the next generation of development-tools that utilize large language models or statically typed languages.@en

Software testing, a critical phase in the software development lifecycle, is often hindered by the time-intensive and costly manual creation of test cases. While automating test case generation could mitigate these challenges, its adoption in the industry has been limited due to difficulties in comprehending the generated test cases. To address this, our study presents an approach for clustering test cases and evaluates its impact on the comprehensibility of test suites through empirical research. Our approach clusters test cases based on their covered objectives, grouping together those with similar attributes to enhance developer understanding. The core of our empirical research evaluates developer agreement with our clustering method and contrasts the comprehensibility of clustered versus non-clustered test suites. Findings suggest a broad agreement among developers in favor of our clustering approach, with clustered test suites facilitating faster software maintenance tasks. Notably, the effectiveness of task completion remained comparable between both suite types. In summary, our research introduces and validates an innovative test case clustering strategy, striving to enhance the comprehensibility of automatically generated test suites.

Docker has been one of the most widely used DevOps tools in the last decade, enabling fast development of personalized services. Indeed, the common practice is to reuse already available containers and customize them based on the developer's needs. DockerHub is the leading platform for uploading and downloading Docker containers. Unfortunately, reusing code and infrastructure exposes developers to security and privacy threats, as the original developer might have had malicious intent to collect sensitive data or create backdoors in a victim's system. The existing literature has raised concerns about this security and privacy threats, and performed a mass vulnerability scans of Docker images. However, currently existing studies are mostly based on static analysis, which has been proved to be insufficient for a complete security assessment.

In this thesis we present a novel framework for the en-masse identification of vulnerabilities in Docker Containers. Additionally, as part of the framework, we document and implement a component which sorts and downloads images based on their popularity, which improves on the current fuzzy-search based state-of-the-art. Using this framework we found vulnerabilities in 2.44% of the containers we scanned. The framework also succeeded in finding novel vulnerabilities, resulting in two new reserved CVE numbers in the social network software Friendica.

Platooning involves organizing a group of vehicles with common attributes into a formation. With the help of Road Side Units, they establish Intelligent Transportation System (ITS). To facilitate communication in this system, one of the prominent communication technologies is Dedicated Short Range Communication (DSRC). In Europe, the European Telecommunication Standards Institute (ETSI) standardizes DSRC as ETSI ITS-G5 in which Decentralized Congestion Control (DCC) is used to prevent and control congestion within a wireless channel. In this work, we determine a list of issues, namely traffic inefficiency and impaired communication, with the conventional formation of a platoon, especially an elongated one. Then, a new paradigm of multi-lane platoons is proposed with a list of modifications based on ETSI DCC to establish a prioritization scheme. These modifications are tailored to the characteristics and regulations of a multi-lane platoon. We also point out the issues of the current adaptive DCC parameters, which motivates us to develop a cross-layer mechanism called dynamic β. This proposal aims to improve the performance of the congestion control mechanism. Our findings demonstrate the possibility of preventing channel congestion by reducing the channel load by 22% at the highest message intensity without compromising the performance of each vehicle. For dynamic β, we manifest a higher channel utilization when employing our proposal, compared to what is achieved with the default ETSI parameters. Message delivery ratio, node capacity, periodicity, and message age are also greatly improved. In short, our work provides safer for more efficient traffic.

Kotlin is a programming language best known for its interoperability with Java, as well as the measurable improvements it offers over it. Since it became Android’s go-to language in 2019, the popularity and impact of Kotlin have risen greatly. Amidst this surge in popularity, the Kotlin developer team is working on a new version of the compiler that introduces sweeping changes to the ecosystem. Traditional compiler testing is a manual and laborious task that requires extensive developer effort and expertise. In an attempt to mitigate this, researchers have invested great resources in developing and perfecting automated compiler testing tools over the last decades. These approaches generate new pieces of code to test the behavior of compilers, which is assessed through differential testing. However, the usage of heuristics as guidance for the generative process is not well understood, and no approach that generates Kotlin code from scratch currently exists. In this thesis, we propose a novel method of enriching standard grammar specifications with language-targeted semantic context that is integrated in the sampling process. We structure generated code hierarchically and use it as the base of an evolutionary computation framework. Within this framework, we introduce two classes of algorithms that are novel to the field of compiler fuzzing, based on syntactic diversity and semantic proximity, respectively. We carry out an empirical analysis spanning 200K generated Kotlin files, which we analyzed through different Kotlin compiler versions. Our results uncovered five previously unreported categories of bugs, which we reported to the Kotlin compiler developer team. The developers verified and replicated our instances on the current re- lease of the Kotlin compiler, and have assigned target release dates for fixes within the current major version of the compiler. The study also provides new insight into the effects of heuristic-specific hyperparameters such as expression simplicity, dissimilarity measurements, and target selection.

On the intuitive level, software testing is important because it assures the quality of the software used by humans. However, ensuring this quality is not an easy task because as the complexity of the software increases, so do the efforts to test it. Search-based software testing is an active research field that develops and explores tools for automatic test case generation. Their work involves using meta-heuristic search optimisation approaches such as evolutionary algorithms
from the evolutionary computation community and applying them to test case generation. The crossover between the evolutionary computation domain and the test case generation one produced DynaMOSA, a state-of-the-art evolutionary algorithm for generating test cases. In an attempt to
produce another well-performing algorithm, this paper performs another crossover between the two communities and creates DynaMOSARVEA - a product of a Reference Vector Guided Evolutionary Algorithm (RVEA) and DynaMOSA.
The conducted empirical study showed that although DynaMOSARVEA does not outperform DynaMOSA, it did outperform RVEA, thus demonstrating the value brought by domain-specific knowledge.

Software testing is an important but time-consuming task, making automatic test case generation an appealing solution. The current state-of-the-art algorithm for test case generation is DynaMOSA, which is an improvement of NSGA-II that applies domain knowledge to make it more suitable for test case generation. Although these enhancements are applicable to other evolutionary algorithms,
no research has been done on how effective other algorithms can function as the base. In this paper, we apply the DynaMOSA modifications to SPEA-II to create a new algorithm, DynaSPEA-II. We conduct an empirical experiment where we evaluate the DynaMOSA enhancements, and directly compare DynaSPEA-II to
DynaMOSA. The algorithms are assessed on a benchmark consisting of 36 diverse JavaScript classes w.r.t. branch coverage. Our results show that adding DynaMOSA enhancements to SPEA-II results in higher coverage in 13.9% of classes, with an average increase of 4.92% for classes where a statistically significant difference was found. DynaSPEA-II performed equally to DynaMOSA, with no statistically significant difference being found between the two.

In recent decades, automatic test generation has advanced significantly, providing developers with time-saving benefits and facilitating software debugging. While most research in this field focused on search-based test generation tools for statically-typed languages, only a few have been adapted for dynamically-typed languages. The larger search-space, generated by the dynamic allocation of types, causes standard search-based algorithms to not be as efficient in this domain and requiring a different approach. Existing algorithms like NSGA-II, MOSA and DynaMOSA have been employed to address this problem, but exploring different approaches may yield better results. That is why this paper proposes a different procedure based on an adaptation of the particle swarm optimization algorithm (PSO). The adaptation was evaluated using
the SynTest framework, showing that DynaMOSA achieves better results than the presented approach, both when comparing the PSO adaptation with and without DynaMOSA features, and when comparing the base DynaMOSA algorithm with PSO adapted to include DynaMOSA ingredients.

Software testing is an important yet time consuming task in the software development life cycle. Artificial Intelligence (AI) algorithms have been used to automate this task and have proven to be proficient at it. This research focuses on the automated testing of JavaScript programs, and builds upon the existing SynTest framework that is the current state of the art, with the Dynamic Many Objective Sorting Algorithm (DynaMOSA) being the best performing AI algorithm for test case generation. DynaMOSA uses the Non-Dominated Sorting Algorithm - II (NSGA-II) as its base algorithm, and adds modifications to it. This paper investigates whether the use of the Pareto Envelope Based Search Algorithm - II (PESA-II) as the base algorithm results in improved performance. The contributions of this research includes a modified PESA-II integrated into the SynTest framework, using inspiration from DynaMOSA. Moreover, we answer the question "How does the modified PESA- II perform compared to DynaMOSA in generating test cases for JavaScript programs?" The performance of the algorithms is measured based on the (branch and method) coverage of the test cases generated for a suite of JavaScript classes. The results show that the modified version of PESA-II outperforms the base version. However, neither manage to outperform DynaMOSA.

Software testing is a laborious job, and accounts for a large portion of software development expenses. Search-based automatic test case generation is an area of research that attempts to remedy this by discovering algorithms suited for generating test cases automatically. In this field, DynaMOSA is a state-of-the-art evolutionary algorithm, which reduces the problem of test case generation to a multi-objective optimization problem, and uses domain knowledge to generate solutions efficiently. In this paper, we adapt Pareto Corner Search Evolutionary Algorithm (PCSEA) for test case generation. Furthermore, we integrate DynaMOSA heuristics into PCSEA, and create a novel algorithm, DynaMOSAPCSEA. We evaluate the test case generation efficacy of PCSEA, DynaMOSA, and DynaMOSAPCSEA by using the JavaScript benchmark provided by SynTest Framework. The results indicate that PCSEA is a feasible algorithm for test case generation, however DynaMOSA heuristics improve its performance only minimally.

Machine learning models are increasingly being used within software engineering for their predictions. Research shows that these models’ performance is increasing with new research. This thesis focuses on models for method name prediction, for which the goal is to have a model that can accurately predict method names. With this thesis, we could create a tool that can suggest method names to software developers, which would assist in improving the quality of the projects.
This research aims to get insight into the robustness vulnerabilities of a method name prediction model. We use a genetic search algorithm that looks for these robustness problems. The main question this thesis tries to answer is to what extent the performance metrics are affected by applying metamorphic transformations to the test set of a trained code2vec model. Besides this, this thesis also proposes an alternative metric called percentage MRR, which might better reflect the robustness of a model. The main idea behind this metric is that it penalizes the prediction certainty of a model instead of penalizing the prediction rank.
To answer this research question, a tool is created that runs a genetic algorithm applying these metamorphic transformations to a dataset that a trained model is then evaluating. With this tool, we conducted 22 genetic search experiments on primary metrics and combinations of metrics to see the trade-offs in the Pareto fronts. The guided search of applying metamorphic transformations on the test set results in an average performance decrease of around 19%. This thesis also compares this drop in performance to the performance decrease a random search algorithm would create. Notably, for every transformer added, the average decrease in performance becomes smaller, and there are transformations, e.g., the if-false-else transformation, that have a bigger effect than others. This thesis concludes that the trained model is not robust against metamorphic transformations and has a significant performance drop.

Cyber-physical systems are complex systems constructed from different independent parts. A self-driving car is an example of a cyber-physical system where independent parts have to come together in order to result in a car that is able to drive by itself. The main challenge is finding failures within the interactions between the independent parts of the self-driving system. In this paper, we present a novel algorithm REWOSA, in order to detect these faults within the self-driving software. With the use of real-world roads extracted from Google Maps combined with a multi-objective genetic algorithm, we develop a new way to generate roads for testing self-driving cars. We evaluate this algorithm against a state-of-the-art multi-objective genetic algorithm using randomly generated roads using two different setups. Our results show that REWOSA is able to generate more failures than the baseline on both the setups, as well as create more complex roads. In return, REWOSA does create a large overhead due to the complexity of the real-world roads. However, this overhead is justifiable as we can detect more faults with the more complex real-world roads.

The continual increase in cyber crime revolving blockchain applications calls for secure blockchain systems and clarity on the division of security responsibilities. This research is an integrated project between two master programmes at the Delft University of Technology: Computer Science and Communication Design for Innovation, and focuses on software testing and security responsibilities.

In this study, we investigate if grammar-based fuzzing, a popular approach for identifying bugs in software, is effective on JSON-RPC systems like blockchain applications Ripple and Ethereum. Furthermore, we evaluate whether we can improve upon traditional grammar-based fuzzing by using evolutionary search.
We introduce GEFRA, a black-box grammar-based fuzzing tool that generates tests for JSON-RPC APIs.
Using a diversity-based fitness function that leverages system feedback, GEFRA is able to effectively guide the search process towards new test cases that obtain additional test coverage.

Additionally, various perspectives on blockchain security responsibilities are investigated. A media content analysis was performed and interviews were conducted with legal and blockchain experts.
News media frequently frame end users as responsible for the prevention of blockchain attacks. While attackers are legally responsible, users are left to deal with the consequences if attackers cannot be found. Responsibilities generally end up with users as decentralisation is the core idea of blockchain. Legislation may be the only solution to define a clear division of responsibilities.

Distributed concurrency bugs (DC bugs) are bugs that are triggered by a specific order of events in distributed systems. Traditional model checkers systematically or randomly test interleavings but suffer from the state-space explosion in long executions. This thesis presents DiscoTest, a testing tool for DC bugs in blockchain consensus algorithms. The tool guides the search for schedules that trigger DC bugs by an evolutionary algorithm (EA). We apply the tool to Ripple's consensus algorithm (RCA) and design and evaluate two representations and fitness functions.

We evaluate the representations on locality, redundancy, and scaling, by using graph edit distance (GED) to calculate the distance between schedules. We find that delay scheduling and priority scheduling are representations that allow variation operators of an EA to modify schedules. To evaluate the performance of the representations and fitness functions, we create a custom bug benchmark for RCA. An empirical comparison on the benchmark shows that delay scheduling with time fitness results in a significantly higher success rate than random search on one bug. Finally, we discover an in-production liveness bug in RCA.

A number of Machine Learning models utilize source code as training data for automating software development tasks. A common trend is to omit inline comments from source code in order to unify and standardize the examples, even though the additional information can capture important aspects and better explain algorithms. We claim that models, utilizing the supplementary data, are able to produce more fluent translations for Automatic Documentation Generation task. We test this by creating two datasets and measuring the performance difference. The results show that there is a slight improvement in translation accuracy when a dataset contains inline comments, with stop words removed. Further research needs to be done to optimize the preprocessing of data and to more accurately detect the scope of inline comments.