Technical debt is a term that describes the consequences of taking shortcuts or quick-and-dirty solutions in the software engineering process, in order to gain short term advantages in the development process of software projects. In this paper, we investigate the technical debt present at the fintech company Adyen, and the developers' experience with technical debt. We investigate their behavior towards testing and refactoring, and give insight into module health with the Delta Maintainability Model. With the conclusions from this research, we propose a novel gamification system called 'Code Karma' to address these technical debt issues, give insight into efforts that increase code quality and motivate developers to improve internal software quality. We found that the Delta Maintainability Model may be used to evaluate module health over time. Additionally, developers believe that 'Code Karma' has a positive influence on the internal quality of the system.

Web APIs are being used for increasingly larger and complex use cases. Right now it can be hard to make sure that what is documented about an API is correct everywhere and to know if a change will have impact on the users of a web API. When details are missing in an API specification users of that API need to make assumptions about how the API works. The creators of the web API also wants to know what users expect from the API. There are two sides to this problem, enforcing that the implementation is actually the same as what is specified, and making it possible to define API specifications as precise as possible. The type system of a programming language is a useful tool for enforcing the struc- ture of an implementation. In this thesis we use a dependent type system to enforce an API specification in the implementation. By using the dependent type system we can de- fine additional, more specific, constraints on the API. These constraints are more specific than constraints expressible in possible research. With this approach we can be sure that the specification and implementation are actually describing the same API. And with the added flexibility we can create a more complete description of web APIs.

Practice is central in mathematics skill acquisition. The practice process can be facilitated by flexible digital exercise systems, supporting personalized learning and providing students with parameterized, open answer exercises containing answer-specific feedback. However, current solutions for authoring these exercises lack efficiency and expressiveness, or rely on unpopular languages for content generation. To address these issues, this work demonstrates a flexible setup for authoring exercises in Jupyter Notebook cells using Python and the available libraries therein and evaluates this setup with expert users and authors of exercises.

The cyber arms race has red and blue teams continuously at their toes to keep ahead. Increasingly capable cyber actors breach secure networks at a worrying scale. While network monitoring and analysis should identify blatant data exfiltration attempts, covert channels bypass these measures and facilitate surreptitious information extraction. The many legitimate uses and widespread availability of DNS, the "phone book" of the internet, make it an attractive protocol for such covert channels. Covert DNS storage channels encode information in the payload of outbound DNS queries.

This thesis aims to assess the effectiveness of using machine learning methods to detect covert DNS storage channels. Our literature survey identified distinct differences in 1) algorithm type, either unsupervised anomaly detection or supervised classification, and 2) the information source for features, either isolated DNS queries or query sequences.

We performed experiments with (Extended) Isolation Forest algorithms for anomaly detection and Random Forests for classification, combined with different feature set compositions to evaluate their relative performance. Payload-only features were derived from isolated queries and behavioral features were extracted from time-based or fixed-length sliding windows over per-domain query sequences. We evaluated our models using a large-scale corporate DNS dataset of real-world proportions and a novel dataset of connection tunneling traffic and simulated credit card exfiltration malware.

We found that the majority of experiments were able to achieve high detection rates of 98.6% or more on a variety of storage channel threats, at low false positive rates. Classification models significantly outperform anomaly detection models on threats seen during training. Evaluation on unseen threats, however, revealed that generalization is difficult, provided the limited set of training threats and showed anomaly detection models more capable at detecting a variety of threats than classification models. We furthermore showed that feature sets with a behavioral component consistently outperform payload­-only features, although our experiments were inconclusive regarding the relative performance between composite feature sets.

Given the prevalence of benign storage channels misusing DNS for legitimate data transfer, we recommend rigorous filtering of training data beforehand to improve model optimization and evaluation. Furthermore, extending the malicious training set with DNS command-and-control (C2) malware is a promising future research direction to improve generalization of classification models.

Software testing is an integral part of the development of embedded systems. Among other reasons, tests are frequently used to ensure that a system meets all the specifications, which is especially important when designing systems for the medical industry. Software changes that have a detrimental impact on a real-time system's performance can accumulate until the systems no longer meet the required performance levels. At this point, fixing accumulated defects can become a costly and complex endeavor. The goal of this thesis is to create a method that can detect changes in performance metrics from a continuous integration pipeline with minimal manual intervention.

To achieve this goal, we have created a novel online, univariate change detection method, which consists of a diverse ensemble of more than 10 existing change detection algorithms. The ensemble is robust against changes in distribution and requires little tuning. The contributions of this work include the proposal of a generic architecture to combine both statistics and decisions from individual algorithms. Related work uses simple majority voting for decision fusion in the ensemble - we demonstrate that an ensemble can benefit from more complex decision fusion, for example using a Random Forest.

Synthetic data and a case study, using a dataset provided by Philips, are used to demonstrate that the overall ensemble is consistently able to outperform the individual algorithms in the ensemble. In addition, unlike the individual algorithms, the ensemble generalizes well to data that is not normally distributed and have not been encountered during training. Compared to the monitoring system for performance metrics that is currently being used by Philips, the ensemble is able to detect 75\% more changes with a 50\% lower false positive rate.

The convenient service offered by credit cards and the technological advances in e-commerce have caused the number of online payment transactions to increase daily. With this rising number, the opportunity for fraudsters to obtain cardholder details via online credit card fraud has also increased. As a result, according to the European Central Bank, billions of Euros are lost due to credit card fraud every year. Since verifying all transactions by hand is infeasible, automated Fraud Detection Systems (FDSs) are needed. Currently, financial institutions create such systems by training machine learning algorithms on transaction data. However, the performance of these systems is obstructed due to a lack of positive (fraud) samples in the collected transaction data. To improve performance, an ideal solution would be to merge data of all institutions and to train an FDS on the resulting data set. However, privacy reasons concerning the sensitive customer information in this data, and security risks associated with transferring data, render this solution unrealistic. Therefore, the need rises for novel protocols that allow financial institutions to collaboratively train FDSs without sharing private data. Previous research in the field of collaborative learning attempts to solve such problems by requiring participants to train local models, which are aggregated into a global model by a trusted central entity. Unfortunately, the vulnerability of these settings to inference attacks restricts their applicability. Inference attacks aim to extract additional secret knowledge from a model. These are especially powerful when performed by participants in a sequential setting, where participants train the same model one after the other following a given order. This is because in this setting participants have white-box access to the model itself and to the data used to train it. Naturally, these attacks are considered a breach of privacy and hinder collaboration. In this work, we propose a novel protocol leveraging secure multi-party computation techniques to prevent inference attacks in a sequential setting. To achieve this, we require participants to jointly determine a training order. While doing so, we ensure participants only receive information on whom to send their data to. This means participants are unaware of whose data they are receiving. With this work, we contribute a practical protocol that is robust against inference and timing attacks to facilitate privacy-preserving sequential collaborative learning. To the best of our knowledge, our work is the first to prevent inference attacks using a secure multi-party selection protocol with overhead of only a few seconds.

A refactoring is a program transformation that improves the design of the source code, while preserving its behavior. Most modern IDEs offer a number of automated refactorings as editor services. The Rename refactoring is the most-commonly applied refactoring and is used to change the identifier of a program entity such as a variable, a function, or a type. Correctly implementing refactorings is notoriously complex and these state-of-the-art implementations are known to be faulty and too restrictive. When developing a new programming language, it is both difficult and time-consuming to implement sound and complete automated refactoring transformations. Language-parametric definitions of refactorings that can be reused by instantiation with the syntax and semantics of a language, allow the development effort of refactorings to be amortized across language implementations. In this thesis, we developed a language-parametric Rename refactoring algorithm that works on an abstract model of a program's name binding structure. We implemented the algorithm in the Spoofax language workbench, building on the language-parametric representation of name binding with scope graphs and using generic traversals in the Stratego transformation language. We evaluated the algorithm with five different languages implemented in Spoofax, which uses both NaBL2 and Statix to declare their static semantics and name binding rules. As a result, Spoofax now provides an automated Rename refactoring that works for any language developed with the language workbench using NaBL2/Statix.

Many types of database management systems exist, but finding the one that is right for a specific use case is becoming increasingly more difficult. Benchmarks allow one to compare various systems, but in a world where distributed DBMSs are increasingly used for mission critical purposes, we find most existing benchmarks neglect fault tolerance and dependability aspects. In this Master’s Thesis, we design a modular and highly extensible framework capable of introducing partial system failures in a distributed database deployment. We also implement a proof-of-concept version of our framework which we use to evaluate the performance of a CockroachDB cluster deployed through Kubernetes, by running the TPC-C benchmark while we inject faults and measure changes in performance. Using this proof-of-concept implementation we demonstrate the faults our system can introduce and find that the impact of our high-level node failures is strongly dependent on the time a node has to perform a graceful shutdown and notify its peers or connected clients.

Computational Thinking (CT) - the process of thinking like a programmer or computer scientist - is a skill that that has the potential to transform the way students learn at educational institutions in different domains and different grade levels. With the increasing integration of CT in classrooms, there is a growing need for CT assessment tools to evaluate the acquisition of CT skills. This research develops a framework for CT assessment that detects user micro-interactions in a university-level self-paced Python beginners course integrated into Jupyter notebooks. The users can improve their learning with the help of feedback via CT dashboards as part of this framework. A user evaluation study was conducted which showed that this framework can be used to improve the acquisition of CT skills via programming. The main contributions of this framework are the mapping between CT skills and user micro-interactions and development of the CT dashboards to help the user self-regulate their learning of programming. The framework developed can be easily integrated into any course that teaches Python programming using Jupyter notebooks and is yet to be extended to other programming courses.

Static Analysis is of indispensable value for the robustness of software systems and the efficiency of developers. Moreover, many modern-day software systems are composed of interacting subsystems written in different programming languages. However, in most cases no static validation of these interactions is applied. In this thesis, we identify the Cross-Language Static Semantics Problem, which is defined as "How to provide a formal and executable specification of the static semantics of interactions between parts of a software system written in different languages?" We investigate current solutions to this problem, and propose criteria to which an all-encompassing solution to this problem must adhere. After that, we present a design pattern for the Statix meta-DSL for static semantics specification that allows to model loosely coupled, composable type system specifications. This pattern entails that the semantic concepts of a particular domain are encoded in an interface specification library, which is integrated in the type system of concrete languages. This allows controlled but automated composition of type systems. We show that, under some well-formedness criteria, the system provides correct results. A runtime, executing composed specifications, is implemented using PIE pipelines for partial incrementality, and integrated in the command-line interface and Eclipse IDE platforms, using the Spoofax 3 Framework. This allows using multi-language analysis in concrete projects. The design pattern, and the accompanying runtime are validated using two case studies. These case studies show that the approach is effective, even in a case where there is an impedance mismatch in the data models of the involved languages.

Active state machine learning algorithms are a class of algorithms that allow us to infer state machines representing certain systems. These algorithms interact with a system and build a hypothesis of what the state machine describing that system looks like according to the behavior they observed. Once the algorithm arrives at a hypothesis, it sends it to a so-called equivalence oracle to be checked. The equivalence oracle returns a counterexample trace describing different behavior between the hypothesis and the real system if one exists. Traditionally, this task is done using a technique called the W-method, which provides certain guarantees about finding a counterexample if one exists, but this is rather slow due to the thorough search it needs to do. In real systems, there are sometimes patterns to the counterexamples to be found during the learning process, which can be exploited to speed up the process of finding new counterexamples. Instead of performing an exhaustive search each time, we first use the already observed patterns to generate new traces to try before falling back to traditional techniques in case no counterexample was found in the first step. This allows us to find counterexamples more quickly than usual, and because we can always fall back on the techniques that do provide guarantees we do not have to sacrifice correctness. In the same amount of time, our method provides an up to 26x increase in the number of states found compared to the W-method, while remaining fully black box and without degrading worst-case performance. We also show that, when not working in a fully black-box scenario, fuzzing can be a successful complementary technique to apply next to active learning. However, we also show that relying on fuzzing alone does not always fully capture the behavior of a system, and when correctness is critical such as in the case of model checking, it needs to be supplemented with additional techniques to result in fully correct models. Finally, we use our home-grown state machine learning library to participate in the 2020 RERS challenge, and apply the techniques we studied so far to attain a perfect score on the sequential LTL track.

Privacy-preserving data aggregation protocols have been researched widely, but usually cannot guarantee correctness of the aggregate if users are malicious. These protocols can be extended with zero-knowledge proofs and commitments to work in the malicious model, but this incurs a significant computational cost on the end users, making adoption of such protocols less likely.

We propose a privacy-preserving data aggregation protocol for calculating the sum of user inputs. Our protocol gives the aggregator confidence that all inputs are within a desired range. Instead of zero-knowledge proofs, our protocol relies on an asynchronous probabilistic hypergraph-based detection algorithm with which the aggregator can quickly pinpoint malicious users. Our protocol is robust to user dropouts and is non-interactive apart from the registration phase. We describe several optional extensions to our protocol for temporal aggregation, dynamic user joins and leaves, and differential privacy. We analyse our protocol in terms of security, privacy, and detection rate. Finally, we compare the runtime complexity of our protocol with a selection of related protocols.

As organizations start to adopt machine learning in critical business scenarios, the development processes change and the reliability of the applications becomes more important. To investigate these changes and improve the reliability of those applications, we conducted two studies in this thesis. The first study aims to understand the evolution of the processes by which machine learning applications are developed and how state-of-the-art lifecycle models fit the current needs of the fintech industry. Therefore, we conducted a case study with seventeen machine learning practitioners at the fintech company ING. The results indicate that the existing lifecycle models CRISP-DM and TDSP largely reflect the current development processes of machine learning applications, but there are crucial steps missing, including a feasibility study, documentation, model evaluation, and model monitoring. Our second study aims to reduce bugs and improve the code quality of machine learning applications. We developed a static code analysis tool consisting of six checkers to find probable bugs and enforcing best practices, specifically in Python code used for processing large amounts of data and modeling in the machine learning lifecycle. The evaluation of the tool using 1000 collected notebooks from Kaggle shows that static code analysis can detect and thus help prevent probable bugs in data science code. Our work shows that the real challenges of applying machine learning go much beyond sophisticated learning algorithms -- more focus is needed on the entire lifecycle.

Since the introduction of the Web, online platforms have become a place to share opinions across various domains (e.g., social media platforms, discussion fora or webshops). Consequently, many researchers have seen a need to classify, summarise or categorise these large sets of unstructured user-generated content. A field related to this task is also known as opinion mining in which various applications have focused on sentiment analysis techniques to classify opinionated documents based on sentiment. More recent, researchers have focused on stance classification to classify opinionated documents based on stance in controversial debates. However, in the case of such controversial debates it would be equally interesting to know the underlying reasons behind a stance in order to truly understand a discussion. We can call these underlying reasons as perspectives. Few have focused on distilling such perspectives from text and in this research we aim to explore the use of an unsupervised model - called joint topic models - to perform the task of perspective discovery. We define perspective discovery on a controversial debate as the process of automatically finding and extracting a structured overview of perspectives from unstructured text. The aim is to quantify how well existing joint topic models can extract human understandable perspectives between and within stances for more fine-grained opinion mining on textual debates. To perform this evaluation we propose an evaluation setup with an extensive user study. This setup focuses on the topic model’s clustering ability of perspectives as well as the human understandability of the topic model’s output. Based on the results we may derive that topic models can discover some of the perspectives from text. Moreover, the results suggest that users are not influenced by their pre-existing stance when interpreting the output of topic models.

Code duplication is a form of technical debt frequently observed in software systems. Its existence negatively affects the maintainability of a system in numerous ways. In order to tackle the issues that come with it, various automated clone detection techniques have been proposed throughout the years. However, the vast majority of them operate using the entire codebase as input, resulting in redundant calculations and undesirable delays when this process is repeated for every new revision of a project. On the other hand, newer incremental techniques address this by storing intermediate information that can be reused across revisions. However, all these approaches are language-specific, utilizing language parsers to generate more sophisticated source code representations, in an attempt to detect more complex types of clones. As a result, less popular languages, for which finding or building a parser is challenging, are unfortunately not supported.

In this study we propose LIICD, a language-agnostic incremental clone detector, capable of detecting exact-match clones. We assess its performance and compare it with a state-of-the-art commercial-grade detector, found within the Software Improvement Group (SIG). Furthermore, we use a similarity estimation technique called Locality Sensitive Hashing (LSH) in an attempt to extend and improve the original approach. Our experiments result in some interesting findings. Firstly, the proposed incremental detector is very efficient and able to scale well for larger codebases. Additionally, it provides a significant improvement compared to a non-incremental commercial-grade detector. Lastly, our LSH-based extension proves to have difficulties matching our original approach's performance. However, future suggestions indicate how the potential of the technique can be further investigated.

Attribution of the malware to the developers writing the malware is an important factor in cybercrime investigative work. Clustering together not only malware of the same family, but also inter-family malware relations together provides more information about the authors and aid further malware analysis work. In this report, previous work which concluded attribution on compiled binaries can be done by a programmer their style is questioned. Given insight on this matter, this report explores new clustering techniques for both static and dynamically derived features from malware binaries. Both methods are complementary as they provide very different types of data. In the static analysis, the data for the similarity comparison is derived from disassembled binaries, while in dynamic analysis the choice was made to record system calls executed by the malware during ecution.
We use a finer granularity than when comparing the data of the complete binaries with each other, such that instead of differences, fine similarities among malware families can be found. Evaluation of clusters is a difficult subject, because of its unsupervised nature and data quality related causes. However, upon manual inspection of the generated clusters, the newly developed clustering methods confirm previously discovered similarities but also find new connections among malware families.

Learning to program is not a easy task, as has become evident from the abundance of research papers concerning the subject. One of the learning barriers of learning a new programming language is understanding their error message, as coding errors have to be resolved before the programmer can run the code or add new functionality to the program. If the error message does not give the (kind of) information the programmer needs or uses terminology the programmer does not understand, it becomes a lot harder to fix the error it is reporting.
This thesis aimed to enhance the understanding about why the error messages of the Python programming language fail to be understood by novice programmers. An experiment was performed where novice programmers constructed how they thought the error messages should look like when encountering errors often made by novice programmers. This thesis also introduces the Error Message Component Framework, a framework which can be used to determine the different components of information an error message contains and the structure of these components. This framework was used to compare the original error message to the custom error messages created by the participants of the experiment, to investigate if the original message matched the novice programmer's information expectations.
This thesis concludes that even though some errors are often made by novice programmers, their resulting error messages may not targeted towards this group based on the terms used in these messages, the component of information these messages contains and the lack of precision of the information.

Modern web information systems use machine learning models to provide personalized user services and experiences. However, machine learning models require annotated data for training, and creating annotated data is done through crowdsourcing tasks. The content used in annotation crowdsourcing tasks like medical records and images might contain some private information which can directly or indirectly identify an individual. The name, age, ethnicity, gender, contact details are examples of private information that directly identifies an individual. Indirect private information relates to the cultural, economic, and social factors of an individual. For instance, the visual cues of religious objects or symbols relate to the religious beliefs of an individual. In this thesis, we study how to minimize the amount of private information extracted from images using a hybrid algorithm which combines machine learning models and crowdsourcing. We also demonstrate that the proposed hybrid algorithm reduces the amount of private information exposed from the image and the cost of using the crowd for detecting private information in the image.

Huge amounts of log data are generated every day by software. These data contain valuable information about the behavior and the health of the system, which is rarely exploited, because of their volume and unstructured nature. Manually going through log files is a time-consuming and labor-intensive procedure for developers. Nonetheless logging information can expose the problematic execution of the software, even though the final outcome seem to be normal. Nowadays the automatic analysis of the log files is crucial for detecting problems, but mainly for understanding how the software behaves, which would be beneficial for the prevention of failures and improvement of the software itself. Towards that direction, this project aims the identifications of unexpected executions of the software and the determination of the root cause behind them. In more details, the expected behavior of the software can be approximated using model inference techniques and the newly incoming observed data can be analyzed to verify if they are conformed by the expected behavior. The conformance checking method that will be used is called replay. The incoming traces will be replayed in the graph, at the point they are not validated, the alignment algorithm will take over. The sequence alignment is performed in three different ways. Two of the methods are looking for the best alignment at a specific radius around the problematic node. Additionally a global alignment technique is implemented, which is based on the famous algorithm by Needleman and Wunsch for DNA sequences. Our goal required the modification of the aforementioned algorithm to not only align two sequences, but a sequence with a tree structured model. Finally the implemented tool provides the visualization of the differences in a way that makes it intuitive for the developers to understand what went wrong. Some additional information are also provided to make the investigation of the "anomaly" easier.

Log data, produced from every computer system and program, are widely used as source of valuable information to monitor and understand their behavior and their health. However, as large-scale systems generate a massive amount of log data every minute, it is impossible to detect the cause of system failure by examining manually this huge size of data. Thus, there is a need for an automated tool for finding system's failure with little or none human effort. Nowadays lots of methods exist that try to detect anomalies on system's logs by analyzing and applying various algorithms such as machine learning algorithms. However, experts argue that a system error can not be found by looking into a single event, but in multiple log event data are necessary to understand the root cause of a problem. In this thesis work, we aim to detect patterns in sequential distributed system's logs that can capture effectively the abnormal behavior. Specifically as a first step, we will apply rule mining techniques to extract rules that represent an anomalous behavior, which potentially in the future may lead to a failure of a system. Except for that step, we implemented a real-time anomaly detection framework to detect problems before they actually occur.
Processing log data as streams is the only way to achieve a real-time detection concept. In that direction we will process streaming log data using a complex event processing technique. Specifically, we would like to combine rule mining algorithms with complex event processing engine to raise alerts on abnormal log data based on automatically generated patterns. The evaluation of the work is conducted on Hadoop's logs, a widely used system in the industry. The outcome of this thesis project gives really promising results, reaching a Recall of 98\% in detecting anomalies. Finally, a scalable anomaly detection framework was build by integrating different systems into the cloud. The motivation behind this is the direct application of our framework to a real-life use case.