Abstract—The integration of unmanned aerial systems (UASs) into existing airspace is imminent as the concept is maturing. U- Space is a concept in which UASs are allowed to fly alongside general aviation aircraft, requiring all aircraft in the airspace to transmit navigation data using radio and network identification. Nevertheless, a consensus is required for the separation in case of conflict amidst the limitations of communication, navigation, and surveillance (CNS) systems. Update rate and probability depict the uncertainty in the communication systems, while spread around the ground truth in position describes the navigation accuracy. Then, BlueSky, an open-source simulator, produces thousands of conflict detection and resolutions to evaluate the CNS limitation on the intrusion prevention rate and loss of separation severity. This paper concludes that higher update rate, probability, and position accuracy lead to safer conflict resolution, and selecting 50 meters as the radius of the protected zone results in a 25-meter intrusion in the worst case. @en

For car traffic it was found that a more crowded area leads to a lower speed and a lower arrival rate. The relation between crowdedness and speed (or arrival rate) can be expressed in a network fundamental diagram, or macroscopic fundamental diagram (MFD). Similar concepts have been shown for pedestrian and train traffic. In this paper, we extend the concept to three spatial dimensions. While simulations have explored some concepts, we present for the first time empirical results of the relation between the crowdedness in the air and the performance of the “network.” We base our results on several months of data of airplanes around Amsterdam Schiphol Airport. Similar to car traffic, we observe a reduction in speeds as the number of airplanes in the area increases. However, even at the highest observed densities, we do not see a reduction in flows. This is because of active and intensive management (based on departure/landing possibilities), comparable to perimeter control in traffic, as well as a minimum airplane speed. This paper introduces an interesting concept of applying a MFD to three-dimensional (3D) spaces. We also show to what extent the performance reduction is caused by speed reduction and to what extent it is caused by less efficient routes. The MFD concept can eventually be used to also manage 3D airspaces for applications with less strict microscopic air traffic management than the current management around airports.@en

Abstract—In the coming decades, drones are expected to operate within urban areas at high volumes, and if implemented suc- cessfully, applications such as infrastructure inspection, medical supply and parcel delivery can be improved by the technology. This poses a challenge: how are these drones to be guided in this highly-constrained airspace? Many existing projects have approached the problem from different angles: some place more importance on the Tactical Layer and thus resolving conflicts in flight, while other research focuses on the Strategic Layer with scheduling or airspace design. While analysis is done on a complete system, with all separation management layers implemented, work remains to be done regarding quantifying how these layers interact, and what positive characteristics of these interactions can be utilised to make the system more efficient, safe, and robust to uncertainties. This paper proposes a framework on which this analysis can be performed. Firstly, lay- ers are investigated independently. A feedback system is proposed, where layer outputs are measured, as is the resulting system performance. For instance, an initial hypothesis is that reducing airspace complexity in the Strategic layer, while accounting for uncertainty, will lead to better overall system performance. This can help with minimising flight times and improving overall safety. Also, manoeuvres performed by the Tactical (in-flight) layer should take this complexity metric into account. The feedback loop approach also proposes that the complexity be fed back to the central planner, and that the Strategic (Pre-Flight) layer should be able to take system status into account when performing planning. @en

Urban air mobility can be a potential solution for urban congestion, and high-level concepts of operations (e.g., UTM, U-space) have been developed with the help of large-scale simulations of multi-agent systems. However, one aspect that should be researched more is the effect of wind on the safety and efficiency of missions in an urban environment. While studies that analyse the potential effect of wind on U-space operations exist, they mostly use constant wind fields, or highly simplified wind models. The study at hand investigates whether medium-fidelity CFD models can be used to predict and match recorded wind data in the city centre of The Hague. Six locations with distinct urban features were chosen, and wind measurements were recorded on two separate days. Using the rooftop wind properties obtained during the measurement sessions, computational fluid dynamics (CFD) simulations were performed within a large urban model of the city. Results indicate that there are large discrepancies between the simulated and measured values. Some wind phenomena observed within the measured wind data were also replicated by the CFD model. Thus, based on the results presented in this work, future research should focus on improving computer city models and wind measurement methods to ensure the development of concepts of operations that maximise the safety and efficiency of future U-space operations. @en

Conventional Air Traffic Control is still predominantly being done by human Air Traffic Controllers, however, as the traffic density increases, the workload of the controllers increases as well. Especially for the area of unmanned aviation, driven by the rise in drones, having human controllers might become unfeasible. One of the methods that is currently being investigated for replacing the conflict resolution task of Air Traffic Control is Reinforcement Learning. As violation of the required separation margins, also called an intrusion, is an event of relatively low frequency, using Reinforcement Learning for this task comes with difficulties that can potentially be attributed to data imbalance. This paper artificially increased the traffic density during the training phase of the Reinforcement Learning method to investigate what the importance is of a balanced data set on the performance of the Reinforcement Learning method. It was found that as the traffic density increased, the Reinforcement Learning methods started to outperform the analytical methods. Beyond this it was found that methods trained at higher traffic densities, but tested at lower traffic densities, outperformed the methods trained at that specific density. This indicates that it might be better to always ensure that the training scenarios are more complex than anticipated during the execution phase, even if that results in unrealistic scenarios.

@en

Reinforcement Learning (RL) is rapidly becoming a mainstay research direction within Air Traffic Management and Control (ATM/ATC). Many international consortia and individual works have explored its applicability to different ATC and U-Space / Urban Aircraft System Traffic Management (UTM) tasks, such as merging traffic flows, with varying levels of success. However, to date there is no common basis on which these RL techniques are compared, with many research parties building their own simulator and scenarios from scratch. This can diminish the value of this research, as the performance of an algorithm cannot be easily verified, or compared to that of other implementations. This hampers development in the long run. The gymnasium library shows for other research domains that this can be solved by providing a set of standardised environments, which can be used to test different algorithms, and compare them to benchmark results. This paper proposes BlueSky-Gym: a library that provides a similar set of test environments for the aviation domain, building on the existing open-source air traffic simulator BlueSky. The current BlueSky-Gym environments range from vertical descent environments, to static obstacle avoidance and traffic flow merging. Built upon the Gymnasium API and the BlueSky air traffic simulator, it delivers an open-source solution for the ATC-specific RL performance benchmark. In the initial release of BlueSky-Gym, 7 functional environments are presented. Preliminary experiments with PPO, SAC, DDPG and TD3 are presented in this paper. Results show stable training is obtained on all of the environments with the default hyperparameters. On some environments, there is a large performance gap, with the on-policy PPO often trailing, but overall no clear algorithm that outperforms others across the board in terms of total reward.@en

The traffic density of small aerial vehicles operating within urban environments is expected to increase significantly in the near future. This urban environment is highly constrained due to being limited to the low-altitude airspace directly above the existing road network. Multiple studies have addressed factors influencing the capacity of urban airspace. These have used simulations of aircraft, yet the empirical nature of these simulations limits their use beyond the specific conditions that have been tested. Analytical models would not have this limitation, but they are only developed for general airspace, while the emergent patterns in constrained urban airspace are different than in general, unconstrained conditions. For instance, queuing and local congestion are patterns that are unique to the heavily-constrained environment. Therefore, in this paper, we derive an analytical model for air traffic in a confined airspace to find the influencing factors for its capacity. By means of a simulation of aerial vehicles, we verify the analytical model and show a relationship between the mean flow rate and mean density in a two-dimensional orthogonal grid network airspace. Results show that the entire airspace can become unstable when the maximum capacity of just one intersection is reached. Furthermore, the maximum airspace density is found to be unaffected by cruise speed. The results demonstrate how the derived analytical model provides an effective tool to predict the effect of several design parameters on the capacity of constrained urban airspace. Moreover, this model can form the basis for further extensions, including the altitude dimension and non-orthogonal or non-four-way intersections.

@en

The number of unmanned aircraft operating in the airspace is expected to grow exponentially during the next decades. This will likely lead to traffic densities that are higher than those currently observed in civil and general aviation, and might require both a different airspace structure compared to conventional aviation, as well as different conflict resolution methods. One of the main disadvantages of analytical conflict resolution methods, in high-traffic density scenarios, is that they can cause instabilities of the airspace due to a domino effect of secondary conflicts. Therefore, many studies have also investigated other methods of conflict resolution, such as Deep Reinforcement Learning, which have shown positive results, but tend to be hard to explain due to their black-box nature. This paper investigates if it is possible to explain the behaviour of a Soft Actor-Critic model, trained for resolving vertical conflicts in a layered urban airspace, by interpreting the policy through a heat map of the selected actions. It was found that the model actively changes its policy depending on the degrees of freedom and has a tendency to adopt preventive behaviour on top of conflict resolution. This behaviour can be directly linked to a decrease in secondary conflicts when compared to analytical methods and can potentially be incorporated into these methods to improve them while maintaining explainability.@en

The use of small aircraft for a wide range of missions in urban airspace is expected to increase in the future. In Europe, efforts have been invested into developing a unified system, called U-space, to manage aircraft in dense very-low-level urban airspace. The Metropolis II project aimed to research what degree of centralisation an air traffic management system should use in such airspace. The paper at hand is a follow-up, and investigates improvements that can be brought to the tactical conflict prevention, detection, and resolution module of such a system in order to harmonise these components with an organic high-density U-space environment. The proposed improvements are: the prioritisation of vertical conflict prevention in intersections, the use of intent in detecting and resolving conflicts, and the use of heading-based manoeuvres in open airspace. Results show that the use of intent information in the conflict detection process, as well as the implementation of suitable tactical prevention procedures, can greatly increase airspace safety. Furthermore, the experiments revealed that the effectiveness of conflict resolution algorithms is highly dependent on the airspace rules and structure. This reiterates the potential for increasing the safety and efficiency of operations within constrained airspace if the tactical separation modules are unified with the other components of air traffic management systems for U-space. @en

Unmanned Aircraft Systems (UAS) Traffic Management (UTM) is an active research subject as its proposed applications are increasing. UTM aims to enable a variety of UAS operations, including package delivery, infrastructure inspection, and emergency missions. That creates the need for extensive research on how to incorporate such traffic, as conventional methods and operations used in Air Traffic Management (ATM) are not suitable for constrained urban airspace. This paper proposes and compares several traffic capacity balancing methods developed for a UTM system designed to be used in highly dense, very low-level urban airspace. Three types of location-based dynamic traffic capacity management techniques are tested: street-based, grid-based, and cluster-based. The proposed systems are tested by simulating traffic within mixed (constrained and open) urban airspace based on the city of Vienna at five different traffic densities. Results show that using local, area-based clustering for capacity balancing within a UTM system improves safety, efficiency, and capacity metrics, especially when simulated or historical traffic data are used.

@en

The interest in urban air mobility as a potential solution for urban congestion is steadily growing. Air operations in urban areas can present added complexity as compared with traditional air traffic management. As a result, it is necessary to test and develop novel airspace designs and rules. As airspace in urban areas is a scarce resource, creating structures and rules that effectively utilise the airspace is an important challenge. This work specifically focuses on layered airspace design in urban operations constrained to fly between the existing buildings. Two design parameters of airspace design are investigated with two sub-experiments. Sub-experiment 1 investigates layer function assignment by comparing concepts from previous research with different layer assignment distributions. Sub-experiment 2 investigates the flight rules of vertical distribution of traffic within the airspace, to determine whether this is best achieved in a static (pre-allocated) or dynamic manner. Both sub-experiments analyse the overall system safety, route duration, and route distance under increasing traffic demand. Results reveal that the importance of cruising airspace is apparent at high densities. Results also shows that the safest layer allocation flight rule depends on the traffic density. At lower densities dynamic rules help to spread traffic locally. However, when the airspace is saturated it is safer to pre-allocate flight heights if achieved uniformly.@en

Future operations involving drones are expected to result in traffic densities that are orders of magnitude higher than any observed in manned aviation. Current geometric conflict resolution (CR) methods have proven to be very efficient at relatively moderate densities. However, at higher densities, performance is hindered by the unpredictable emergent behaviour from neighbouring aircraft. Reinforcement learning (RL) techniques are often capable of identifying emerging patterns through training in the environment. Although some work has started introducing RL to resolve conflicts and ensure separation between aircraft, it is not clear how to employ these methods with a higher number of aircraft, and whether these can compare to or even surpass the performance of current CR geometric methods. In this work, we employ an RL method for distributed conflict resolution; the method is completely responsible for guaranteeing minimum separation of all aircraft during operation. Two different action formulations are tested: (1) where the RL method controls heading, and speed variation; (2) where the RL method controls heading, speed, and altitude variation. The final safety values are directly compared to a state-of-the-art distributed CR algorithm, the Modified Voltage Potential (MVP) method. Although, overall, the RL method is not as efficient as MVP in reducing the total number of losses of minimum separation, its actions help identify favourable patterns to avoid conflicts. The RL method has a more preventive behaviour, defending in advance against nearby neighbouring aircraft not yet in conflict, and head-on conflicts while intruders are still far away.

@en

Over the next decades, it is expected that the number of unmanned aerial vehicles (UAVs) operating in the airspace will grow rapidly. Both the FAA (Federal Aviation Administration) and the ICAO (International Civil Aviation Organisation) have already stated that aircraft operating autonomously or beyond their operators’ line of sight are required to have detect and avoid capabilities. At higher traffic densities these avoidance manoeuvres can, however, lead to instabilities within the airspace, causing emergent patterns that lead to knock-on effects that can harm the safety of the operations. It might be possible to formulate a cost function that encapsulates global safety, rather than individual safety, stimulating both safety and stability. One method that lends itself for optimizing such a cost function is cooperative Multi-Agent Reinforcement Learning (MARL). It has been demonstrated that MARL can be used for optimization in both competitive and cooperative (or even mixed) environments, however, when applied in a completely decentralized manner, stability issues often arise. It is therefore proposed to investigate the application of MARL for a well known centralized domain, ATC for manned aviation. This doctoral paper breaks down the proposed research project into 4 independent phases that individually contribute to the knowledge of applying MARL for ATC.@en

Current estimates show that the presence of unmanned aviation is likely to grow exponentially over the course of the next decades. Even with the more conservative estimates, these expected high traffic densities require a re-evaluation of the airspace structure to ensure safe and efficient operations. One structure that scored high on both the safety and efficiency metrics, as defined by the Metropolis project, is a layered airspace, where aircraft with an intended heading are assigned to a specific altitude layer. However, a problem arises once aircraft start to vertically traverse between these layers, leading to a large number of conflicts and intrusions. One way to potentially reduce the number of intrusions during these operations is by using conventional conflict resolution algorithms. These algorithms however have also been shown to lead to instabilities at higher traffic densities. As recent years have shown tremendous growth in the capabilities of Deep Reinforcement Learning, it is interesting to see how well these methods perform in the field of conflict resolution. This research investigates and compares the performance of multiple Soft Actor Critic models with the Modified Voltage Potential algorithm during vertical manoeuvres in a layered airspace. The final obtained performance of the trained models is comparable to that of the Modified Voltage Potential algorithm and in certain scenarios, the trained models even outperform the MVP algorithm. Overall, the results show that DRL can improve upon the current state of conflict resolution algorithms and provide new insight into the development of safe operations.@en

Current predictions on future drone operations estimate that traffic density orders of magnitude will be higher than any observed in manned aviation. Such densities redirect the focus towards elements that can decrease conflict rate and severity, with special emphasis on airspace structures, an element that has been overlooked within distributed environments in the past. This work delves into the impacts of different airspace structures in multiple traffic scenarios, and how appropriate structures can increase the safety of future drone operations in urban airspace. First, reinforcement learning was used to define optimal heading range distributions with a layered airspace concept. Second, transition layers were reserved to facilitate the vertical deviation between cruising layers and conflict avoidance. The effects of traffic density, non-linear routes, and vertical deviation between layers were tested in an open-source airspace simulation platform. Results show that optimal structuring catered to the current traffic scenario improves airspace usage by correctly segmenting aircraft according to their flight routes. The number of conflicts and losses of minimum separation was reduced versus using a single, uniform airspace structure for all traffic scenarios, thus enabling higher airspace capacity.

@en

Deep Reinforcement Learning has seen more usage in the field of Air Traffic Control over the last couple of years. As the number of aircraft in a given sector of airspace is not constant, there is a need for methods to be invariant to the number of agents in the system. Often this is done by making a selection of the aircraft that will be included in the state, which introduces human biases. Another option that has been used is Recurrent Neural Networks to process the entire sequence of aircraft present. These methods however are sequence-dependent and can give different results depending on the order that the aircraft are given, which is undesirable. Methods that solely rely on attention mechanisms, such as transformers, allow sequential data to be processed in a sequence-invariant manner by using multi-head attention mechanisms. However, because traditional Transformers operate on individual tokens, this does not allow for relative state information to be encoded into the hidden state. This paper shows that by performing a transformation operation on the key and value tokens, it is possible to use Transformers on relative states, at the cost of a factor (N-1) additional attention computations, where N is the number of agents in the system. This adaptation allows relative state Transformers to obtain significantly higher performance than standard Transformers. The results also showed that using attention mechanisms to construct the initial observation vector out of a total of 20 agents results in similar, but slightly lower, performance to handcrafted observation vectors, without requiring manual selection of the important agents. Future research should investigate whether additional changes to the attention mechanisms and their training can result in higher performance.@en

In order to enable the safe and efficient integration of Unmanned Aerial Vehicles into very low level airspace, current day research focuses on the development of new traffic services and procedures. One of these is the geovectoring protocol, which aims to reduce traffic complexity by setting limits on the allowed ground speed, course, and vertical speed. A geovector can be used to increase the capacity of an airspace by lowering the conflict rate. However, problems emerge when performing avoidance maneuvers in geovectored airspace, as the limits are ignored in this process. A powerful conflict resolution algorithm is the Modified Voltage Potential (MVP). This paper proposes an extension to the MVP algorithm, based on Velocity Obstacle theory. Making use of an alternative horizontal conflict resolution maneuver which respects the geovector, five resolution strategies are defined with different priority settings for the separate limits. The performance of these strategies is compared to pure MVP on geovector, safety, and stability measures, making use of fast-time simulations in a corridor airspace. All geovector resolution strategies show improvements on the ability to perform conflict resolution maneuvers within the geovector limits, while having marginal effects on the overall airspace safety level. It is recommended to further investigate the performance of the geovector resolution strategies for other types of airspace, to verify whether the observed reduction in conflict rate from the geovectors can be reinforced by the resolution strategies.@en

Current predictions for future operations with drones estimate traffic densities orders of magnitude higher than any observed in manned aviation. Such densities call for further research and innovation, in particular, into conflict detection and resolution without the need for human intervention. The layered airspace concept, where aircraft are separated per vertical layer according to their heading, has been widely researched and proven to increase traffic capacity. However, aircraft traversing between layers do not benefit from this separation and alignment effect. As a result, interactions between climbing/descending and cruising aircraft can lead to a large increase in conflicts and intrusions. This paper looks into ways of reducing the impact of vertical transitions within the environment. We test two reinforcement learning methods: a decision-making module and a control execution module. The former issues a lane change command based on the planned route. The latter performs operational control to coordinate the longitude and vertical movement of the aircraft for a safe merging manoeuvre. The results show that reinforcement learning is capable of optimising an efficient driving policy for layer change manoeuvres, decreasing the number of conflicts and losses of minimum separation compared to manually defined navigation rules.

@en

The rate of urbanization is expected to continue increasing. This has led to an interest in using drones and air taxis for urban transportation in place of the current methods, which often lead to road congestion. In most places urban air operations will happen above buildings. However, in many cities with large skyscrapers it may not be efficient to fly above buildings as it would add travel distance. For these cases, aircraft will have to operate in constrained airspace (above roads and between buildings). There is still a knowledge gap for operating in constrained very low-level urban airspace. Most studies attempt to improve the safety in constrained airspace with strategic or tactical conflict resolution. But this may not be enough to ensure safety in highly-dense urban environments. The restriction of heading manoeuvres by buildings substantially limits the solution space for conflict resolution. Therefore, conflict prevention with airspace design can be an important tool for improving airspace safety. In a layered airspace, turn layers can be used so that turning aircraft do not create bottlenecks for cruising aircraft that may be behind it. However, merging conflicts can occur when these turning aircraft attempt to re-enter cruising layers. These are typical in both orthogonal (New York) and non-orthogonal (Paris) street networks. Non-orthogonal street networks can also create merging conflicts because it is not always possible to segment cruising aircraft at intersections. This work will propose two conflict prevention doctoral research experiments that aim to reduce merging conflicts. The first will use three different layering techniques to reduce merging conflicts created by turn layers. The second will focus on merging conflicts that are typical of non-orthogonal networks. @en

Package delivery via autonomous drones is often presumed to hold commercial and societal value when applied to urban environments. However, to realise the benefits, the challenge of safely managing high traffic densities of drones in heavily constrained urban spaces needs to be addressed. This paper applies the principles of traffic segmentation and alignment to a constrained airspace in efforts to mitigate the probability of conflict. The study proposes an en-route airspace concept in which drone flights are directly guided along a one-way street network. This one-way airspace concept uses heading-altitude rules to vertically segment cruising traffic as well as transitioning flights with respect to their travel direction. However, transition flights trigger a substantial number of merging conflicts, thus negating a large part of the benefits gained from airspace structuring. In this paper, we aim to reduce the occurrence of merging conflicts and intrusions by using a delay-based and speed-based merge-assist strategy, both well-established methods from road traffic research. We apply these merge assistance strategies to the one-way airspace design and perform simulations for three traffic densities for the experiment area of Manhattan, New York. The results indicate, at most, a 9–16% decrease in total number of intrusions with the use of merge assistance. By investigating mesoscopic features of the urban street network, the data suggest that the relatively low efficacy of the merge strategies is mainly caused by insufficient space for safe manoeuvrability and the inability for the strategies to fully respond and thus resolve conflicts on short-distance streets. @en