Offensive AI for Directory Enumeration

Abstract

Web Vulnerability Assessment and Penetration Testing (Web VAPT) is an important cybersecurity practice that thoroughly examines web applications to uncover possible vulnerabilities. These vulnerabilities represent potential security gaps that could severely compromise the web applications' integrity and functionality if exploited by malicious entities.
One of the attacks employed in the Web VAPT process is the Directory Brute-Forcing Attack. This attack aims to identify hidden directories and files not adequately secured in a web application that contain sensitive information or critical functionalities. The attack methodology involves sending many requests of possible directories or files to the target web application, where brute-force generation of requests is performed using a wordlist. Due to its brute-force nature, this attack methodology often results in enormous quantities of requests sent for a small amount of successful discoveries.
With AI's quick progress and diffusion, the paradigm of Offensive AI emerges, where AI-based technologies are employed in traditional cyber attacks to make them more sophisticated and effective.
This research explores whether AI can enhance the standard directory enumeration process. We propose two novel attack methodologies for performing directory brute-forcing attacks that leverage probability and Language Models (LM).
Our experiments - conducted on a testbed consisting of around 1 million URLs from various domains of web applications (academic institutions, hospitals, government agencies, and business corporations) - demonstrate the superiority of our approaches over the standard brute-force attacks.
In particular, the LM-based attack results in an average discoveries increase of 969%, and the probabilistic attack is more efficient at sending successful requests in the early stages of attacks in more than 94% of cases.