Pavement undergoes a fast deterioration process either due to the damages induced by weather conditions, an increase in traffic flow and load, or passive factors like aging of infrastructure. Thus requiring periodic rehabilitation measures to maintain the condition of the underlying asset. Since damages on the asphalt road, impose economic setbacks and concern for the users, governmental authorities are looking for a proactive approach to detect and classify distresses in their "early" stages. As a reason, governmental authorities like the Province of Zuid-Holland (PZH) yearly inspect the road network, which was optimal until now, but as the traffic flows are increasing and weather conditions are worsening, a new approach is required to mitigate the need for a frequent, cost-effective and reliable inspection method. Modern data sources such as smartphones are the biggest data generators. Having an intriguing number of sensors and in-built features, governmental and private authorities are just starting to acknowledge the potential of such crowd-sourced data generators. In this research data-types like vibration and imagery were gathered and synthesized to assess the efficiency and accuracy of the 3 data-driven models. A 7-step methodology was implemented to build all the machine learning models. At first, vibration data was gathered to detect road anomalies and predict the International Roughness Index (IRI), by building a Random Forest decision tree. Secondly, a Convolution Neural Network (CNN) was constructed and utilized to classify pavement surface distresses. The last model, is a next step towards autonomously distinguishing the classified distress with its severity ranking. A Deep Neural Network (DNN) called EnDec (Encoder & Decoder) architecture was built and trained on the Dutch supercomputer called "cartesius", by utilizing multi-threading opportunities, to objectively segment the given pavement surface distress.