Understandable Log-Based Anomaly Localisation through Inter-Host Distances

Abstract

While artificial intelligence (AI) has undeniably ushered numerous solutions across various fields, the growing belief that AI can solve all problems overshadows their lack of transparency that comes along. Understanding how decisions are made and what has led to the output is crucial in critical systems to ensure accountability and trust.

This research proposes a complementary method leveraging inter-host distances that localise the outlying hosts, logs and the time frames, which require more advanced analysis. By relying on a variant of a prominent statistical method in the field of authorship attribution - Burrows Delta - the approach enhances transparency in identifying deviating hosts, logs and time frames. Hence, the proposed solution offers an understandable complementary method that preserves integrability by being a log-based method while enabling understandable pinpointing of the specific hosts, logs and time frames that warrant further advanced analysis. By providing insights into the behaviour of the hosts over time, a temporal summarisation for security analysts is provided, relaxing their need to go through all the log files to understand the hosts' behaviour.

The results show that a complementary method based on the textual content of the metadata of the logs provides alternative insight into the activities of the hosts than the attributes. Moreover, the behaviour defined by the proposed method requires less extensive lookup than the behaviour defined by attributes. The inter-host distances based on the textual content allow understandable localisation of the host behaviour over time. Hence, this research provides an understandable method that will summarise the behaviour of the hosts over time, which enables the localisation of the logs requiring more advanced, in-depth analysis, and thereby reducing the amount of logs security analists need to consider during a compromise.