Knowledge on adversaries during military missions at sea heavily influences decision making, making identification of unknown vessels an important task. Identification of surrounding vessels based on visual data offers an alternative to AIS information (Automatic Identification System), the current standard in vessel identification, which can be spoofed. One visual approach employs human expertise and manually identifies vessels guided by a ship catalog. In order to minimize or potentially eliminate human error and performance limitations, there is strong interest in developing an automated vessel classification pipeline. One such pipeline is currently being developed at TNO, capable of classifying over 500 separate classes. A crucial part of the classification pipeline is retrieving an accurate contour of a vessel from a digital image.
To address this important challenge, this thesis proposes an advanced deep learning pipeline to automatically segment the vessel image into background (e.g. sky and sea) and the object of interest (a vessel). Deep learning models based on Fully Convolutional Neural Networks (FCNs) have achieved high performance on the task of semantic segmentation. Several networks such as CRF-RNN, PSPNet, DeepLab and Mask R-CNN are employed to determine a baseline performance. We will focus on identifying the cause of poor or failing segmentations and aim to construct a robust network capable of handling these challenges. By sampling disturbances, caused by ship distance and camera noise, augmented data sets are built to tune networks to input from on-site images. Additionally, experiments are done to evaluate the influence of different levels of disturbances.
Previous approaches implementing the CRF-RNN network achieved top 1 and top 5 classification accuracies of 31.1% and 44.0% respectively. Employing the DeepLab network, trained to convergence on artificial noise augmented data, we report top 1 and top 5 accuracy of 68.9% and 88.8% respectively. Additionally, implementing an ensemble of classifiers, performance is increased to 73.0% and 91.7% for top 1 and top 5 accuracy respectively. This best result is comparable to the classification results with human annotated ship silhouettes. The human performance accuracy is 73.4% on top 1, and 91.3% on top 5 classification performance. Finally, we show that training on a collection of different levels of image disturbances results in a network that is robust against increasing disturbance in images, while retaining performance on clean images.

In hyperspectral (HS) imaging, for every pixel a spectrum of wavelengths is captured. These spectra represent material properties, i.e. the spectral signatures. So, classification of HS imagery is based on material properties. This thesis describes a framework to perform pixelwise classification of HS images of a fixed scene subject to varying ambient conditions. TNO has recorded HS images over the course of one year, every hour between sunrise and sunset. Therefore, this data set is subject to a range of lighting, weather and seasonal conditions, degrading the recorded data. Traditionally, atmospheric models are used to correct for these effects, recovering spectral information. In this work a Fully Convolutional Neural Network (FCN) is trained to perform the segmentation task which also learns to correct for ambient conditions, eliminating the need of implementing an atmospheric model or applying image normalization. A single FCN (U-Net) is implemented to solve a five-class segmentation problem, distinguishing broad leaf trees, grass, sand, asphalt and artificial grass. To start training a neural network, the training data set requires a corresponding ground truth. A sparsely annotated mask is designed which fits images covering the entire recording period. In single-scene segmentation, annotating using a sparse mask is a quick method which allows for moving object borders. Furthermore, it avoids inclusion of mixed pixels. In order to reduce computational load, the training data set is formed by many small patches taken from the original HS images. Consequently, the network is trained with local spectral-spatial information. Training the standard U-Net proves to be limited to training data sets under relatively constant ambient conditions. In order to further enhance generalization over seasons, the network weights are rearranged so that a similar number of weights is maintained. This network is essential for training a complex training data set, as it is able to extract more informative features. The standard U-Net trained with a simple training data set (i.e. relatively constant ambient conditions) achieves an accuracy A > 94% for both sunny and rainy test images irrespective of time of the day. However, this model is valid for a limited period of time. The customized U-Net trained with a complex training data set (i.e. highly varying ambient conditions) yields segmentations with an accuracy ranging between 86 and 93%. This model is valid for a longer period of time, covering multiple seasons. So the experiments show that there is a trade-off between segmentation accuracy and duration of model validity, which is controlled by network weight arrangement.