3D building models play an important role in many real-world applications. Different models are suitable for different application scenarios based on their levels of detail. LOD3 models with facade details are crucial for many applications, such as virtual reality and urban simulation. Currently, 3D building models with lower LOD are largely available, but the number of LOD3 models is very limited. Most LOD3 reconstruction methods depend on manual operation, which is very time-consuming. How to automatically reconstruct the detailed facade for building models has remained a problem in computer vision. The problem can be seen as an image processing problem, but how to convert the 2D results into 3D smoothly should also be considered. 

In this project, we proposed a method to automatically reconstruct the detailed building models based on the Faster R-CNN. The method starts from a set of street view images, and the results are models with facade elements. A 3D point cloud can be extracted from the images using SfM and MVS, and the camera parameters can also be recovered. We take advantage of the high-quality facade images and parse the facades to detect their bounding boxes. The bounding boxes can match pretty well with the rectangular shape of the facade elements. The 2D facade elements can be added to the 3D building model based on the camera parameters. The process is very efficient and automatic. The regularity of the facade elements will be reserved, making the result more convincing. Our method includes four main steps:  (1) coarse model reconstruction, (2) facade image selection and rectification, (3) facade element detection and regularization, and (4) detailed facade reconstruction.

Experiment results show that our method can produce reliable building models with facade details for many different situations. It can work for both the multi-face building blocks and the street side buildings. Our test shows that the window detection performance is pretty good. The object detection is extremely fast, and the whole pipeline is lightweight and efficient. In theory, the method can also be extended to reconstruct large-scale city models, which means it has broad application prospects.

As a method that can accurately represent 3D spatial information, point cloud visualisation for indoor environments is still a relatively unexplored field of research. Our client for this project, the Dutch National Police, requested a variety of potential solutions for visualising (unfamiliar) indoor environments that can be viewed by both external command centres, and internal operations units. Currently, unknown interior layouts (or layouts that are different in practise to what is stated on paper) can have serious, sometimes even life-threatening, consequences in time-sensitive situations. This project uses a game engine to directly visualise point cloud data input of indoor environments. The primary aim is to find ways of clearly communicating a point cloud of an environment to a layman viewer through intuitive visualisations, to aid decision-making in high-stress moments. The final product is a variety of visualisation concepts, hosted within a game engine in order to allow users to navigate throughout (part of) a building, and customise certain interaction features. To aid the layman viewer, various interpretation methods (e.g. cartography) are considered. The Unreal Engine 4 (UE4) project was designed and developed based on the requirements given by Dutch Police, and consisted of 4 modules: data preprocessing, render style, functional module, and User Interface (UI). An indoor point cloud dataset is used for the implementation, while corresponding mesh and voxel models are also respectively generated and evaluated as reference objects. The implemented software product is evaluated based on a Structured Expert Evaluation Method and finally our project result demonstrates that point cloud has unique advantages for visualisation of indoor environments especially in pre-processing efficiency, detail level, and volume perception.