Transforming the global energy sector from fossil-fuel based to renewable energy sources is crucial to limiting global warming and achieving climate neutrality. The decentralized nature of the renewable energy system allows private households to deploy photovoltaic systems on their rooftops. However, inconsistent data on installed photovoltaic (PV) systems complicate planning for an efficient grid expansion. To address this issue, deep-learning techniques, can support collecting data about PV systems from aerial and satellite imagery. Previous research, however, lacks the consideration for ground truth data-specific characteristics of PV panels. This study aims to implement a semantic segmentation model that detects PV systems in aerial imagery to explore the impact of area-specific characteristics in the training data and CNN hyperparameters on the performance of a CNN. Hence, a U-Net architecture is employed to analyze land use types, rooftop colors, and lower-resolution images. Additionally, the impact of near-infrared data on the detection rate of PV panels is analyzed. The results indicate that a U-Net is suitable for classifying PV panels in high-resolution aerial imagery (10 cm) by reaching F1 scores of up to 91.75% while demonstrating the importance of adapting the training data to area-specific ground truth data concerning urban and architectural properties.