Roofing material classification is becoming pivotal in urban decision-making, supporting processes like asbestos mapping, disaster preparation, and urban heat island detection. However, research has mainly focused on a restricted set of materials. Additionally, many studies rely on expensive multi- or hyper-spectral imagery and often use outdated or ineffective classification methods, overlooking the transformative capabilities of advanced deep learning and data fusion. This thesis explores using a convolutional neural network for pixel-level classification of Dutch buildings by merging standard aerial images with LiDAR data, enabling detailed mapping and addressing concerns about classification efficacy relative to image- and object-level methods.

A framework was devised to generate a new semantic segmentation dataset with over 15.5 million pixels from 200 randomly selected images nationwide, covering eight distinct materials. To facilitate material identification in unfavourable lighting conditions, true-colour aerial imagery from the BM5 dataset was combined with rasterised features extracted from the national point cloud (AHN4), specifically reflectance, slope, and planar point density. Additionally, a quasi-normalised elevation model (nDRM) was employed, based on the corresponding digital surface model and median roof elevation of buildings in each scene, as provided by the 3DBAG dataset. The research was further investigated using the DeepLabv3+ semantic segmentation architecture with a ResNet-18 backbone, and the model was trained end-to-end on the generated dataset. In this context, a novel stratified splitting algorithm and weighting scheme to combat class imbalance in the training subset were introduced.

After thorough hyperparameter tuning, we achieved a 64.68% mean intersection over union on the test subset. Membranes and gravel outperformed almost every other study. However, there were notable confusion and omission errors with light-permitting surfaces and metal. Further testing of pixel-wise material maps' generalization to different LoDs of the 3DBAG considerably decreased gross errors. However, it might overlook some minor original predictions, thus not improving overall performance notably. Generally, LoD1.2 was inadequate for modelling multi-material roofs of different heights. While LoD1.3 improved this, it still missed small roof sections, unlike LoD2.2, which also had more outliers. Additionally, an ablation study on the LiDAR-derived component of the new dataset showed that removing slope and nDRM reduced performance by 10.31% and 8.61%, respectively, while density had the least impact. All ablated features were semantically linked, suggesting they should be combined into a single dataset.

The thesis showcases the relevance of pixel-based classification with DL and data fusion, providing resources for future research and indicating areas for dataset expansion and improved annotation.

The reconstruction of 3D city models has garnered significant interest in recent years. However, the majority of existing reconstruction methods primarily focus on LOD2 models, while LOD3 model reconstruction often relies on manual labor, and the primary data sources are street view images. This research aims to advance this field by reconstructing LOD3 models through the addition of windows and doors to existing LOD2 models, thereby maximizing the utility of available 3D building models, as well as the accurate addition of windows and doors. This research innovatively utilizes aerial oblique images as the data source for extracting building openings and employs 3D BAG LOD2.2 models as the basic 3D building structures. The 3D facades are projected onto the 2D aerial image space using perspective projection and registration is employed on the projection facade and oblique aerial images. Subsequently, Mask R-CNN is employed to detect and extract the building openings from these projections. Following the extraction, the layout of the openings within the same facade is optimized in terms of both size and position. Lastly, the relative positions of the openings on the facade images are combined with the 3D coordinates of the corresponding facade to calculate the positions of the openings in 3D space. This information is then integrated into the LOD3 model, resulting in a more detailed and accurate representation of the buildings.
This approach successfully reconstructs the final LOD3 model in CityJSON format, which passes the val3dity validation. By effectively utilizing existing 3D building models, this approach conserves a considerable amount of computational resources required for reconstruction. The simplicity and high level of automation of this approach make it a promising solution for reconstructing large-scale LOD3 buildings, leading to more accurate and detailed large 3D urban models.

Three-dimensional (3D) city models are of great significance and are high in demand. They can be used for various useful applications such as urban planning, visibility analysis and estimating the solar irradiation and energy demand of buildings throughout the day. Nowadays, Light Detection And Ranging (LiDAR) sensors are one of the most commonly used technologies for the acquisition of cheap, fast, dense, and reliable 3D point cloud datasets. At the same time, increasing attention has been focused lately on the utilization of these 3D point cloud datasets for the reconstruction of 3D city models. The 3D representation of a scene in the form of a 3D point cloud facilitates a variety of analysis tasks like object recognition, segmentation, and classification, which are an important prerequisite for many building reconstruction methods. Thus, having an accurate classifier that can automatically assign 3D points a respective semantic class label is of utmost importance as it can significantly reduce the time and cost required to analyse 3D scenes. Training machine learning and deep learning algorithms to perform this task has been the focus of many recent scientific works that provide promising results and insights for future work.
This thesis attempted to create an accurate Random Forest Classifier for LiDAR point cloud datasets using the Actueel Hoogtebestand Nederland 3 (AHN3) dataset as training data. The aim was to assist building reconstruction methods, and for this reason, only three semantic class labels are assigned to the points by the classifier, namely ground, building and other. Multiple experiments were conducted to test how large the training dataset needs to be, what features should be included and what point density the input point cloud needs to have for the classifier to perform well. Tests were also made using the DALES dataset to evaluate the performance of the classifier for different datasets and environments. Moreover, the time and memory required to train and test the various Random Forests models and the evaluation metrics of the results were stored as benchmarking research to provide insights and guide future work.
The classification approach that was used consists of five major steps. First, the input point cloud is uniformly sampled and then spherical local neighbourhoods of the points are computed in multiple scales. Height and eigen-based features are then extracted for each local neighbourhood, and the best subset of features is then used to train the Random Forest Classifier. Finally, each point of the input point cloud is assigned to the same class as its nearest neighbour in the sampled point cloud. A comparative analysis shows that the performance of our final Random Forest model stands in parallel with that of other available, more complex, deep learning algorithms if we take into consideration its simplicity and efficiency.