Passage re-ranking is a fundamental problem in information retrieval, which deals with reordering a small set of passages based on their relevancy to a query. It is a crucial component in various web information systems, such as search engines or question-answering systems. Modern approaches for building re-ranking systems rely on neural language models such as BERT, or its derivatives, to create dense indexes for the target document corpus. While such approaches bring significant performance gains compared to classical lexical re-rankers, they have the disadvantage of increased memory costs.

A family of methods that can be used to reduce the memory footprint of a dense index is called vector quantization. Vector quantization algorithms usually rely on a combination of clustering and space manipulation operations to perform a lossy compression of the dense index at the expense of index performance. While vector quantization is widely used for first-stage retrieval, its use in the context of re-ranking is underexplored. To this end, this thesis evaluates the effectiveness of product quantization, a well-known vector quantization method, on single-vector dual-encoders, specifically TCT-ColBERT and Aggretriever. In addition to this, we show how linear interpolation of sparse scores can be leveraged to improve the performance of quantized dense indices with negligible costs to the memory footprint or speed. Last but not least, we propose WolfPQ, a learnable quantization method aimed at further improving quantization for re-ranking by bridging the gap between the objective functions used in training product quantization and re-ranking systems, respectively.

With the increase of machine learning applications in our every-day life, high-quality datasets are becoming necessary to train accurate and reliable models. This research delves into the factors that contribute to a high quality dataset and examines how different dataset metrics affect the performance of machine learning models particularly focusing on Graph Neural Networks (GNNs) Tabular Transformers and Large Language Models (LLMs). The metrics, under scrutiny include graph sparsity, missing data cells, modularity and text length. Various datasets are adjusted to assess how these metrics impact model performance.

The results of the experiments reveal that sparse graphs can preserve relational information. However increasing density does not necessarily lead to improved performance due to noise interference. The models demonstrated accuracy and low error rates in the presence of significant missing data indicating their ability to handle incomplete information effectively and generalize well based on imputation strategies and structural design. Higher modularity was found to aid in capturing patterns. Introduced complexity that could potentially hinder performance. Notably text length emerged as a factor influencing model performance by offering contextual details.

These insights show the significance of considering attributes when designing machine learning models for intricate predictive tasks. Through experimentation and optimization of these metrics we can enhance model resilience and accuracy for applicability, in real world scenarios.

The first step of many cyber attacks is the reconnaissance phase. One of many reconnaissance methods employed by adversaries is internet-wide scanning, which
probes the entire internet to find which hosts have open ports. These scans are practically
impossible to detect by a firewall or Intrusion Detection System if an attacker chooses to
distribute their scan on multiple hosts. Many of these scans embed a fingerprint in their
packets, which can easily be detected if they are known. Previous studies have developed
an algorithm that is able to identify these fingerprints, but they were not able to identify
fingerprints for large portion of their data. This study proposes an iterative approach
using stratified sampling, in order to see how this affects accuracy. An experiment showed
the algorithm is able to identify fingerprints for sets of packets that make up less than
0.5% of all packets, and less than 0.0001% of sources. Analysis of the fingerprinted groups
indicated that these groups are not part of a collaborative scanner, but hold for the same
fingerprint by coincidence.

While LLMs are proficient in processing textual information, integrating them with other models presents significant challenges.
This study evaluates the effectiveness of various configurations for integrating a large language model (LLM) with models capable of handling multimodal data.\\

We explore the advantages of using pre-trained LLMs for generating text embeddings and the benefits of fine-tuning LLMs for specific tasks. Our investigation includes various fine-tuning strategies, such as Low-Rank Adaptation (LoRA), prompt tuning, and full fine-tuning, applied to both smaller and larger language models. Additionally, we analyze different training setups, including sequential and cascaded training of LLMs and downstream architectures. Our comparative analysis evaluates the performance and cost-effectiveness of these methods. The findings indicate that while full fine-tuning achieves the best results, LoRA offers a practical balance between computational efficiency and model performance. We also highlight the correlation between increased LLM size and corresponding increases in cost and performance.

Port-Scanning is a popular technique that helps detect open ports to connect to on the internet, with both benign and malicious applications. While methods have been developed to detect scans coming from one source, adversaries have started to distribute their scans among multiple hosts. Detecting these collaborative scanners, however, is a difficult task, and no established method has emerged yet. In this paper, we propose a method to cluster scanning groups from network telescope data based on the assumption that scanners working together will have similar behavioral features. An evaluation framework based on address space coverage and overlap is introduced and the performance of two clustering algorithms, HDBSCAN and DBSCAN, is compared. A clear indication was found that only relying on behavioral features and applying unsupervised clustering techniques can result in incomplete groups or clusters with more than one group inside, with HDBSCAN generally performing better than DBSCAN. The effectiveness of the evaluation framework is also discussed, as some of the identified groups indicated overlaps, but manual analysis reveals that they have the potential to belong to a single party.

Deep Learning models can use pretext tasks to learn representations on unlabelled datasets. Although there have been several works on representation learning and pre-training, to the best of our knowledge combining pretext tasks in a multi-task setting for relational multimodal data has not been done before. In this work, we implemented 4 pretext tasks on top of a framework for handling relational multi-modal data and evaluated them based on two datasets. We first identified the best-performing masking strategy for pretext tasks that require masking. Then, we compared different combinations of the pretext tasks based on self-supervised metrics as a proxy for the quality of the representation learned. The results reveal that masking values by replacing from the column's empirical distribution yields 4.6\% and 4\% higher accuracy on each dataset respectively than replacing them with a fixed value. We also found that different combinations of pretext tasks, even with different numbers of tasks, converge to marginally different values and MoCo further reduces this difference. Our findings imply that the number of pretext tasks can scale efficiently allowing for a more diverse representation to be learned.

This research investigates the effectiveness of combining Feature Tokenizer Transformer (FTTransformer)[6] with graph neural networks for anti-money laundering (AML) applications. We explore various fine-tuning techniques, including LoRA[9] and vanilla fine-tuning, our baseline FTT architecture. Using the IBM AML dataset [1], we compare the performance of different models and fine-tuning approaches. Our results indicate that FTT alone do not outperform GNN’s and careful configuration is required when working with datasets of Multi-Modality. This work contributes to the development of more efficient and accurate methods for detecting financial fraud patterns.

The substantial amount of tabular data can be attributed to its storage convenience. There is a high demand for learning useful information from the data. To achieve that, machine learning models, called transformers, have been created. They can find patterns in the data, learn from them, and improve their predictive abilities based on that learning experience. There are also tabular transformers for tabular data. In order to attempt to increase the predictive performance of the transformers, we have combined them with graph neural networks (GNNs), which are again machine learning models, which work on graph data by learning information from the nodes and the edges. A graph representation of the dataset is created and input into the graph neural network. The architecture that fuses these two machine learning models is a more complex machine learning model that combines the transformer and the GNN. The aim is to increase the predictive ability of the model for values from the table or to predict whether an edge in the graph exists, which represents whether a transaction between two users exists. We have built the architecture using certain types of a tabular transformer and a graph neural network, FT-Transformer and GINe respectively, and the next step is to try modifying this architecture by using different models, and different ways of using these layers, for example how many copies we are creating of it. This has the potential to be a versatile model than can be used for different kinds of datasets. We have seen notable improvement in performance when using a different GNN, PNA. The transformer ResNet also shows to be on a similar or slightly better performing level than FT-Transformer when not combined with a GNN. GraphSage in the fused model underperforms significantly due to its weakness to capture simple graph structures.

Port scanning is a technique often used by adversaries to detect vulnerable services running on a machine. There are defense mechanisms in place that can detect fast, single-source port scanning, but one of the ways to remain hidden is to distribute the scan between multiple hosts. These distributed groups of machines can divide the address space and collaboratively scan the whole Internet within minutes and remain relatively hidden.
This paper proposes a simple method to detect these collaborative scanners based on the TCP/IP header and demonstrates its efficiency. It also tracks these scanners for a longer period and describes their behavior and how they develop over time. This includes the infrastructure they utilize, the specific ports they target, and additional relevant details. This perspective has not been previously explored in the academic literature and we find it to be important such that defenders get a better understanding of the threats they are facing.

Detecting distributed scans is crucial for understanding network security threats. This research uses an algorithmic approach to identify collaborative ZMap scanning activities in the network telescope data from TU Delft. ZMap is a high-speed network scanner capable of scanning the entire IPv4 address space. The main research question centers on creating an algorithm for detecting these distributed scans. The research method includes analyzing network telescope data, examining ZMap packets and modifying the set cover algorithm to detect collaborative ZMap scans. Key contributions include adapting the set cover algorithm to find sources that perfectly cover the entire destination address range without overlaps, which is a unique feature of ZMap scans. Results indicate that this method effectively identifies coordinated scanning activities. It is concluded that utilizing network telescope data and an adapted version of the greedy set cover algorithm significantly improves the detection of distributed scanning operations using ZMap.

This research investigates the effectiveness of combining Feature Tokenizer Transformer (FTTransformer)[6] with graph neural networks for anti-money laundering (AML) applications. We explore various fine-tuning techniques, including LoRA[9] and vanilla fine-tuning, on our baseline FTT architecture. Using the IBM AML dataset [1], we compare the performance of different models and fine-tuning approaches. Our results indicate that FTT alone do not outperform GNN’s and careful configuration is required when working with datasets of Multi-Modality. This work contributes to the development of more efficient and accurate methods for detecting financial fraud patterns.

This paper investigates the effectiveness of various clustering algorithms in detecting collaborative Internet scanning groups. The packet dataset used is collected from TU Delft's network telescope, and is aggregated into scanning sessions and analyzed using K-Means, Hierarchical Clustering, Density-Based Spatial Clustering of Applications with Noise (DBSCAN), Clustering Using Representatives (CURE), and Bradley-Fayyad-Reina (BFR). This paper also introduces an evaluation framework based on five degrees of certainty to assess the likelihood that a cluster is collaboratively scanning. The findings indicate that DBSCAN consistently outperforms other methods in identifying collaborative scanning groups, while CURE shows superior performance to BFR, K-Means, and Hierarchical Clustering. It is hoped that these insights help provide a strong foundation for enhancing network security through improved detection of collaborative scanning behaviors.

This thesis addresses the growing demand for faster and more reliable data transfer solutions in data-intensive applications, with a strong emphasis on scalability and flexibility. The research centers on the design and implementation of a high-speed data transfer system that utilises PCIe over fiber optics. The main objective is to explore how PCIe technology can be effectively extended over long distances using fiber optics, while maintaining high throughput and low latency, essential for supporting the demands of modern data-intensive environments.

The design and implementation of the system were approached methodically, with careful selection of components and a focus on optimising the system architecture for performance. Detailed experimentation was conducted to evaluate the system's capabilities, resulting in successful data transfer rates of up to 58.02 Gbps using a Gen3 x8 PCIe configuration over a 10-meter fiber link. These findings confirm the viability of PCIe over fiber as an effective solution for high-speed data transfer, while also identifying specific areas where further improvements could be made.

Future work could explore the adoption of newer PCIe generations and the optimisation of critical parameters, such as Maximum Payload Size (MPS). Additionally, there is potential to enhance the system's performance by investigating optical connectors capable of supporting sideband signals. These advancements would further improve the system's adaptability and efficiency across a wide range of application scenarios.

Graph Neural Networks (GNNs) have achieved state-of-the-art performance in various applications due to their ability to capture complex structural relationships within graph data. However, their inherent black-box nature poses significant challenges to model interpretability, particularly in the context of link prediction tasks. This thesis investigates the explainability of GNNs for link prediction, a critical task in domains such as social network analysis and recommendation systems. We propose an efficient extension of the Zorro explainer, originally designed for node classification, to address the unique challenges of explaining link predictions. Our method leverages top similar nodes for initialization and incorporates techniques to accelerate the greedy search process, ultimately producing binary node and feature masks as explanations. Additionally, we introduce a novel Edge-level RDT-Fidelity metric to better evaluate explainers that generate edge masks, complementing existing metrics like RDT-Fidelity and Sparsity. Experimental results on the Cora and PubMed datasets demonstrate the superior performance and efficiency of our Zorro extension. This work contributes to the growing field of GNN interpretability by providing new methodologies and evaluation frameworks specifically tailored for link prediction.