The internet traffic is constantly rising nowadays due to the significant increase of the devices connected to the Internet. As a consequence, many cyber risks have arisen. Cybercriminals are trying to exploit the vulnerabilities of these devices to cause damage and gain profit. Monitoring the network traffic and detecting such threats has become essential in order to keep safe systems that are connected to the Internet. The powerful properties of state machines and the sequential nature of the network traffic data, makes them an interesting and promising solution for the implementation of an intrusion detection system.

The goal of this thesis is to implement a new state-merging heuristic which will speedup the state machine building procedure without a significant loss on the quality of the model, and use it to detect malicious host on network traffic data. The new state-merging heuristic is utilizing the Locality-sensitive Hashing concept to store the future traces of each state and simplify the consistency check for the merge of two states. The network traffic data used are in the NetFlow format, and they are encoded and converted into traces in order to build the state machine model and measure its performance. The state machine built is modeling a malicious behavior and used to classify other hosts.

We show that the models built can effectively detect the malicious hosts, with its performance being comparable to the one of a state-of-the-art model. At the same time, the time needed to build the model is much less when compared to the time needed by other state-merging heuristics.