Urban Energy Modeling (UEM) provides a comprehensive approach to urban planning, helping to create sustainable, resilient, and energy-efficient cities that meet the needs of current and future generations. The key inputs for UEM methodologies and tools are the geometry of the building stock and its thermophysical properties. In the Netherlands, the 3DBAG provides the building stock geometry, while the thermophysical properties can be approximated using energy consumption estimates specific to each residential building type from the IEE project TABULA. However, a challenge arises as open data on residential building types at a national level is currently not readily available, necessitating the development of a method to infer this information from other accessible data sources.

In response to similar successful studies, this thesis also focuses on utilising machine learning to address this challenge. Support Vector Machine (SVM) and Random Forest (RF) algorithms were tested and compared. These algorithms were trained using data on the residential building types obtained from the Rijssen-Holten energy testbed and EP-online, with the latter requiring preprocessing to obtain the relevant information. Additionally, 25 features derived from cadastral data and building geometry underwent a selection process to identify the essential features for accurate classification of building types.

Eight models were trained and applied across eight case studies, containing subsets of the Netherlands representative for the whole country. The combined results were analyzed to determine necessary features, required data, and the most suitable machine learning approach for this research.

The results revealed that features such as adjacency to other buildings, width, and volume in LoD2.2 correlated the most with Dutch residential building types. However, critical features like the number of storeys, the presence of an open porch, or galleries were not available as open data, even though they directly relate to the definition of certain residential building types.

This thesis presented successful models, demonstrating accuracies between 61.1% and 98.5%, and balanced accuracies ranging from 51.6% to 94.2%. Importantly, performance differences were observed in various case studies, particularly in distinguishing accuracy between multi-dwelling and single-dwelling houses. Despite the longer tuning and training time, the suitability and accuracy of the RF models generally outperformed the SVM models.

These findings highlights the capacity of machine learning to attain robust classification outcomes for the Dutch building stock when trained on representative datasets. Nevertheless, it is emphasized that the accuracy of results is contingent on data quality, and difficulties may arise in scenarios involving intricate buildings with multiple components and ambiguous classification rules.

Generative AI is developing rapidly and has shown great potential in generating and editing images with text prompts. It has achieved partial success in the challenging 3D model edits of common objects. However, there is a lack of attention in the building domain, which already faces certain limits in the 2D space. 3D building models automatic edit has wide application potential, especially for renovation, concept comparison, and large-scale scenes. Therefore, the thesis aims to edit the building models effectively through texture creation and geometric changes with generative AI by evaluating the existing 3D edit pipelines and putting forward practical solutions based on them.

The thesis first examines the current methods. Latent-paint, Text2Tex and X-Mesh are chosen as the representative existing explicit representation (mesh) based pipelines for their relatively satisfying performance in common objects and the different ideas contained. Modifications are made based on them to improve output quality from the aspects of the control module and edit module. For the first aspect, attempts include adding the view specification text prompt to the original X-Mesh and using the image as the control for X-Mesh. Image control X-Mesh is also further experimented with placing higher attention on the input image view and editing the key semantic parts of the building separately. For the latter aspect, modifications include freezing sampled vertices geometry and combining X-Mesh and Text2Tex.

Results show that Latent-Paint mainly creates texture with only major colour and lacks details. Text2Tex generally generates fine textures for inputs with richer depth information. X-Mesh creates textures and edits geometry jointly. It can improve the fidelity of the input mesh to some extent but suffers from noise problems. Image control X-Mesh generates more realistic results and outperforms the text control pipeline. Combining Text2Tex with X-Mesh takes advantage of Text2Tex, which can generate smoother and more realistic textures compared to X-Mesh in some cases. The combination is especially recommended for low-fidelity mesh of small or medium size. The user study also shows that the combination of X-Mesh and Text2Tex generates the most favoured results while image control X-Mesh ranks second. There are still limitations in geometric deformation scope, fidelity, generalizability and computational efficiency in the proposed 3D edit pipelines. The thesis succeeds in proving the application potential of 3D edit pipelines in the building domain and obtaining high-quality results by modifying existing methods.

Multiview Stereo (MVS) reconstruction techniques have made significant advancements with the development of deep learning. However, their performance often deteriorates in low-light conditions, where feature extraction and matching become challenging. Traditional image enhancement solutions are insufficient for MVS tasks in low illumination, relying on manual adjustments. We introduce an end-to-end MVS framework incorporating a diffusion-based image enhancement algorithm with MVS to build an end-to-end framework for improving the performance of MVS in low-light conditions. This integration improves color rendering and visualization of 3D reconstructions and slightly enhances geometric shapes. Our method uses a feature adapter to integrate the enhanced images from the Low-light Diffusion model into CasMVSNet, refining the feature maps in poorly lit environments. Validation on the DTU and Tanks and Temples datasets demonstrates our model’s robustness and generalizability across various lighting conditions and MVS pipelines, including GeoMVSNet and MVSNet. Our approach simplifies the training process by requiring only the training of an adapter rather than a multi-view image enhancement model, underscoring the effectiveness of incorporating image enhancement into learning-based MVS frameworks for low-light conditions.

In this thesis we present a new idea to objectively assess reconstruction algorithms. Because it is not feasible to completely scan a high-detailed ground truth mesh of large urban objects, the performance of the reconstructed meshes can therefore not be measured objectively. To solve this, we present a new mesh evaluation methodology that can more objectively assess the quality of the generated mesh based on a low detailed ground truth mesh. We achieved this by creating a synthetic dataset based on low-detailed models of large urban buildings and using this as a ground truth mesh and input data for the reconstruction algorithms. Thanks to our new methodology, we were able to compare the output mesh and ground truth mesh using a wireframe model of the meshes. This allows us to give a more objective score to the results without having to look at entire model, which is the usual method. The results of this thesis show that the new methodology has potential to be used for creating a new benchmark, and it opens a new door to using more readily available objects that could not be used before.

The accuracy and comprehensiveness of 3D city models have become increasingly important for applications like monitoring, sustainability evaluation, disaster management, and urban planning. However, creating accurate and complete 3D models is challenging. Traditional methods, such as photogrammetry and Light Detection and Ranging (LiDAR), often face issues like time-consuming processes and data gaps caused by occlusion. Additionally, these methods typically produce discrete models that fail to capture all the information from the original objects, limiting the ability to reconstruct small structures.

In this thesis, we explore the potential and characteristics of enhancing 3D models for urban areas using implicit neural representation. This method offers generalizability, ensures no void areas, and provides more detailed information when using multi-modal inputs. The datasets used for model training include Actueel Hoogtebestand Nederland 3 (AHN3), 2019 Luchtfoto Beeldmateriaal, 3D Basisregistratie Adressen en Gebouwen (BAG), and Basisregistratie Grootschalige Topografie (BGT). The city center of Eindhoven is used for training and testing, and the city center of Rotterdam serves as a test dataset.

The method involves learning location-dependent latent codes from raw point cloud and orthophoto, and adjusting the probability of space occupancy based on point clouds sampled from the 3D city model. A decoder is used to calculate the probability of existence for any point in the 3D space from the continuous field. Proper sampling allows us to derive a continuous Digital Surface Model (DSM) with unlimited resolution from the neural network.

The results show a high degree of accuracy: the generated DSM for the training area in Eindhoven demonstrated a median absolute error of 0.484 m overall. Accuracy for building areas was recorded at 0.798 m, and for terrain, it was 0.302 m. The model also displayed robust generalization capabilities, with an accuracy of 0.336 m in the Eindhoven test area and 0.23 meters in Rotterdam. Additionally, the model's ability to fill voids was confirmed through both visual inspection and quantitative evaluations. This research underscores the potential of implicit neural representation for generating detailed DSMs and effectively filling no-data voids.

Street view imagery (SVI) is one of the largest (growing) resources in urban analytics. A global close-up of the urban environment, if you will, which is rich in (untapped) information such as floor count. Floor count is useful in many applications, from improving energy consumption calculations to creation of 3D city models without elevation data. So far, efforts to extract floor count from SVI are mainly approached as a classification problem with the use of convolutional neural networks (CNNs). Limitations of this approach include the need of large (manually annotated) datasets, and uncertainty how these models learn to count storeys. Therefore, we aim to develop a method that can be trained on available datasets and determine floor count in a more explainable manner.

In order to make the floor count determination method more transparent, we mimic the row-wise counting of storeys as humans do: by vertically parsing a column of windows (and occasional door). Façade parsing is a common computer vision task that we can solve with deep learning. In this work, we employ the Mask R-CNN framework, that is trained on publicly available datasets, for the detection and segmentation of windows and doors. Then, the vertical distribution of detected / segmented windows and doors is estimated by computing the kernel density estimation function. The floor count is extracted by finding the number of maxima in the function, as the maxima represent the dense areas of windows and doors on a horizontal axis (i.e. storeys). To improve the results, an automatic image rectification is added as pre-processing step that enforces the regularity and repetitive occurrence of windows and doors. The full pipeline thus consists of three stages: 1) automatic image rectification, 2) window and door detection/ segmentation with Mask RCNN, 3) floor count estimation via maxima finding on the kernel density estimation (KDE) function. In addition, a small "wild" dataset was created that contains a higher variability in floor count, image quality and architectural styles, which better reflect real world SVI than existing façade datasets.

The floor count performance of the full pipeline was evaluated on the Amsterdam Facade (subset), ECP, TRIMS and "wild SVI" datasets. Since floor count annotations were missing, these are manually added. For detection-based data, the best results are an accuracy of 83% and a mean absolute error (MAE) of 0.17. For normalised segmentation-based data, the best results are an accuracy of 80% and a MAE of 0.20. Considering the method is still at its infancy, the results are promising. With further improvements in the pipeline and addition of automatic façade acquisition, the approach can contribute in large scale extraction of floor count information from SVI. To encourage further development, the pipeline prototype, dataset and floor count annotations are open source and will be released on https://github.com/Dobberzoon/Facade2Floorcount.

This thesis introduces a Learned-Based Multi-View Semantic Stereo method, addressing the limitations of traditional and learned-based Multi-View Stereo (MVS) techniques in reconstructing reflective and low-textured regions, particularly prevalent in 3D models of buildings. Traditional methods lack completeness, while learned-based methods struggle with accuracy. Focusing on enhancing 3D models of buildings, this research integrates semantic information into the existing deep learning architecture for depth prediction, specifically CasMVSNet, to guide the reconstruction process. Three key strategies are employed: first, the incorporation of semantic maps into the network through a multi-modal approach; second, the introduction of a multi-modal refinement module at the end of the CasMVSNet model to improve the initial output depth maps; and third, the introduction of two new loss terms designed to enforce varying degrees of smoothness on specific semantic categories. Experimental results, conducted on the DTU dataset, demonstrate a significant enhancement in accuracy at the point cloud level while maintaining the completeness of the reconstructed models. Validation and generalization on the ETH3D dataset show consistent patterns. This research showcases the potential of integrating semantic guidance in 3D reconstruction of buildings, advancing the field of computer vision.

Piping and Instrumentation Diagrams (P&IDs) are graphical representations utilized in chemical engineering plants. Due to confidentiality reasons and legacy drawings, these diagrams are sent in PDF format. Piping engineers need to make a material take-off (MTO), a document containing all the components of a P&ID from these drawings. Today, this is done manually, which proves to be time-consuming and laborious. A piping engineer spends approximately 36 hours per 10 P&IDs, with an average of 500-1000 P&IDs per project. Given the expertise and value of process engineers, this manual counting process incurs substantial costs and a repetitive workload. Consequently, there is a growing motivation to automate this process.

In response, this thesis introduces an innovative deep learning model, PandID-Net, designed specifically for P&IDs. PandID-Net uniquely integrates symbol detection, line detection, and text recognition into a single model, diverging from previous methods that relied on separate models and rule-based techniques. It is the first method that uses deep learning for the line detection task in P&IDs. This all in one approach not only simplifies the processing pipeline but also enhances computational efficiency in detecting and pinpointing symbols, lines, and text, as well as their interrelationships.

The optimal configuration of PandID-Net is found by an ablation study where the performance of individual components is tested in isolation. This optimized configuration is then evaluated and benchmarked against a prior study by Paliwal et al. on the same dataset. PandID-Net achieves a performance in F1 scores of 92.89 and 94.48 for line detection and keypoint detection respectively

Style transfer is a recent field in the development of deep neural networks, which allows for the style from one image to be transferred onto another image. This has been well-researched for 2D images, but transferring style onto 3D reconstructed content can still be further developed. Being able to style a 3D reconstruction would allow users to recreate anything in the real world, such as a chair, with any style they see fit. Where other methods use texture-based approaches which often create low quality geometry and appearance, or
use radiance fields which style a whole scene instead of just the 3D reconstructed object, we have developed a method which styles an implicit surface.
We achieve this by using Implicit Differentiable Renderer (IDR), which trains, using masked
images as input, two neural networks that learn the geometry and appearance. Rendered views of the object are styled using 2D neural style transfer (NST) methods, and the style information is used to further train the appearance network to display the given style. With Masked deferred back-propagation we are able to optimize the appearance renderer, which is normally trained on only patches of the rendered image to save memory, while using style
transfers designed for full-resolution images. We showcase different results from our method using different 3D reconstruction datasets and style images, and showcase how to implement a user-created dataset. We carry out
extensive tests on what effects different parameters have on the final result. Comparing our results to similar 3D style methods demonstrates that our method performs equally well in achieving faithful style transfer, while having the benefits of creating high quality geometry and only styling the reconstructed surface.

3D building models play an important role in many real-world applications. Different models are suitable for different application scenarios based on their levels of detail. LOD3 models with facade details are crucial for many applications, such as virtual reality and urban simulation. Currently, 3D building models with lower LOD are largely available, but the number of LOD3 models is very limited. Most LOD3 reconstruction methods depend on manual operation, which is very time-consuming. How to automatically reconstruct the detailed facade for building models has remained a problem in computer vision. The problem can be seen as an image processing problem, but how to convert the 2D results into 3D smoothly should also be considered. 

In this project, we proposed a method to automatically reconstruct the detailed building models based on the Faster R-CNN. The method starts from a set of street view images, and the results are models with facade elements. A 3D point cloud can be extracted from the images using SfM and MVS, and the camera parameters can also be recovered. We take advantage of the high-quality facade images and parse the facades to detect their bounding boxes. The bounding boxes can match pretty well with the rectangular shape of the facade elements. The 2D facade elements can be added to the 3D building model based on the camera parameters. The process is very efficient and automatic. The regularity of the facade elements will be reserved, making the result more convincing. Our method includes four main steps:  (1) coarse model reconstruction, (2) facade image selection and rectification, (3) facade element detection and regularization, and (4) detailed facade reconstruction.

Experiment results show that our method can produce reliable building models with facade details for many different situations. It can work for both the multi-face building blocks and the street side buildings. Our test shows that the window detection performance is pretty good. The object detection is extremely fast, and the whole pipeline is lightweight and efficient. In theory, the method can also be extended to reconstruct large-scale city models, which means it has broad application prospects.