Reinforcement Learning (RL) is rapidly becoming a mainstay research direction within Air Traffic Management and Control (ATM/ATC). Many international consortia and individual works have explored its applicability to different ATC and U-Space / Urban Aircraft System Traffic Management (UTM) tasks, such as merging traffic flows, with varying levels of success. However, to date there is no common basis on which these RL techniques are compared, with many research parties building their own simulator and scenarios from scratch. This can diminish the value of this research, as the performance of an algorithm cannot be easily verified, or compared to that of other implementations. This hampers development in the long run. The gymnasium library shows for other research domains that this can be solved by providing a set of standardised environments, which can be used to test different algorithms, and compare them to benchmark results. This paper proposes BlueSky-Gym: a library that provides a similar set of test environments for the aviation domain, building on the existing open-source air traffic simulator BlueSky. The current BlueSky-Gym environments range from vertical descent environments, to static obstacle avoidance and traffic flow merging. Built upon the Gymnasium API and the BlueSky air traffic simulator, it delivers an open-source solution for the ATC-specific RL performance benchmark. In the initial release of BlueSky-Gym, 7 functional environments are presented. Preliminary experiments with PPO, SAC, DDPG and TD3 are presented in this paper. Results show stable training is obtained on all of the environments with the default hyperparameters. On some environments, there is a large performance gap, with the on-policy PPO often trailing, but overall no clear algorithm that outperforms others across the board in terms of total reward.@en

Urban air mobility can be a potential solution for urban congestion, and high-level concepts of operations (e.g., UTM, U-space) have been developed with the help of large-scale simulations of multi-agent systems. However, one aspect that should be researched more is the effect of wind on the safety and efficiency of missions in an urban environment. While studies that analyse the potential effect of wind on U-space operations exist, they mostly use constant wind fields, or highly simplified wind models. The study at hand investigates whether medium-fidelity CFD models can be used to predict and match recorded wind data in the city centre of The Hague. Six locations with distinct urban features were chosen, and wind measurements were recorded on two separate days. Using the rooftop wind properties obtained during the measurement sessions, computational fluid dynamics (CFD) simulations were performed within a large urban model of the city. Results indicate that there are large discrepancies between the simulated and measured values. Some wind phenomena observed within the measured wind data were also replicated by the CFD model. Thus, based on the results presented in this work, future research should focus on improving computer city models and wind measurement methods to ensure the development of concepts of operations that maximise the safety and efficiency of future U-space operations. @en

Abstract—In the coming decades, drones are expected to operate within urban areas at high volumes, and if implemented suc- cessfully, applications such as infrastructure inspection, medical supply and parcel delivery can be improved by the technology. This poses a challenge: how are these drones to be guided in this highly-constrained airspace? Many existing projects have approached the problem from different angles: some place more importance on the Tactical Layer and thus resolving conflicts in flight, while other research focuses on the Strategic Layer with scheduling or airspace design. While analysis is done on a complete system, with all separation management layers implemented, work remains to be done regarding quantifying how these layers interact, and what positive characteristics of these interactions can be utilised to make the system more efficient, safe, and robust to uncertainties. This paper proposes a framework on which this analysis can be performed. Firstly, lay- ers are investigated independently. A feedback system is proposed, where layer outputs are measured, as is the resulting system performance. For instance, an initial hypothesis is that reducing airspace complexity in the Strategic layer, while accounting for uncertainty, will lead to better overall system performance. This can help with minimising flight times and improving overall safety. Also, manoeuvres performed by the Tactical (in-flight) layer should take this complexity metric into account. The feedback loop approach also proposes that the complexity be fed back to the central planner, and that the Strategic (Pre-Flight) layer should be able to take system status into account when performing planning. @en

Abstract—The integration of unmanned aerial systems (UASs) into existing airspace is imminent as the concept is maturing. U- Space is a concept in which UASs are allowed to fly alongside general aviation aircraft, requiring all aircraft in the airspace to transmit navigation data using radio and network identification. Nevertheless, a consensus is required for the separation in case of conflict amidst the limitations of communication, navigation, and surveillance (CNS) systems. Update rate and probability depict the uncertainty in the communication systems, while spread around the ground truth in position describes the navigation accuracy. Then, BlueSky, an open-source simulator, produces thousands of conflict detection and resolutions to evaluate the CNS limitation on the intrusion prevention rate and loss of separation severity. This paper concludes that higher update rate, probability, and position accuracy lead to safer conflict resolution, and selecting 50 meters as the radius of the protected zone results in a 25-meter intrusion in the worst case. @en

Conventional Air Traffic Control is still predominantly being done by human Air Traffic Controllers, however, as the traffic density increases, the workload of the controllers increases as well. Especially for the area of unmanned aviation, driven by the rise in drones, having human controllers might become unfeasible. One of the methods that is currently being investigated for replacing the conflict resolution task of Air Traffic Control is Reinforcement Learning. As violation of the required separation margins, also called an intrusion, is an event of relatively low frequency, using Reinforcement Learning for this task comes with difficulties that can potentially be attributed to data imbalance. This paper artificially increased the traffic density during the training phase of the Reinforcement Learning method to investigate what the importance is of a balanced data set on the performance of the Reinforcement Learning method. It was found that as the traffic density increased, the Reinforcement Learning methods started to outperform the analytical methods. Beyond this it was found that methods trained at higher traffic densities, but tested at lower traffic densities, outperformed the methods trained at that specific density. This indicates that it might be better to always ensure that the training scenarios are more complex than anticipated during the execution phase, even if that results in unrealistic scenarios.

@en

For car traffic it was found that a more crowded area leads to a lower speed and a lower arrival rate. The relation between crowdedness and speed (or arrival rate) can be expressed in a network fundamental diagram, or macroscopic fundamental diagram (MFD). Similar concepts have been shown for pedestrian and train traffic. In this paper, we extend the concept to three spatial dimensions. While simulations have explored some concepts, we present for the first time empirical results of the relation between the crowdedness in the air and the performance of the “network.” We base our results on several months of data of airplanes around Amsterdam Schiphol Airport. Similar to car traffic, we observe a reduction in speeds as the number of airplanes in the area increases. However, even at the highest observed densities, we do not see a reduction in flows. This is because of active and intensive management (based on departure/landing possibilities), comparable to perimeter control in traffic, as well as a minimum airplane speed. This paper introduces an interesting concept of applying a MFD to three-dimensional (3D) spaces. We also show to what extent the performance reduction is caused by speed reduction and to what extent it is caused by less efficient routes. The MFD concept can eventually be used to also manage 3D airspaces for applications with less strict microscopic air traffic management than the current management around airports.@en

The interest in urban air mobility as a potential solution for urban congestion is steadily growing. Air operations in urban areas can present added complexity as compared with traditional air traffic management. As a result, it is necessary to test and develop novel airspace designs and rules. As airspace in urban areas is a scarce resource, creating structures and rules that effectively utilise the airspace is an important challenge. This work specifically focuses on layered airspace design in urban operations constrained to fly between the existing buildings. Two design parameters of airspace design are investigated with two sub-experiments. Sub-experiment 1 investigates layer function assignment by comparing concepts from previous research with different layer assignment distributions. Sub-experiment 2 investigates the flight rules of vertical distribution of traffic within the airspace, to determine whether this is best achieved in a static (pre-allocated) or dynamic manner. Both sub-experiments analyse the overall system safety, route duration, and route distance under increasing traffic demand. Results reveal that the importance of cruising airspace is apparent at high densities. Results also shows that the safest layer allocation flight rule depends on the traffic density. At lower densities dynamic rules help to spread traffic locally. However, when the airspace is saturated it is safer to pre-allocate flight heights if achieved uniformly.@en

The number of unmanned aircraft operating in the airspace is expected to grow exponentially during the next decades. This will likely lead to traffic densities that are higher than those currently observed in civil and general aviation, and might require both a different airspace structure compared to conventional aviation, as well as different conflict resolution methods. One of the main disadvantages of analytical conflict resolution methods, in high-traffic density scenarios, is that they can cause instabilities of the airspace due to a domino effect of secondary conflicts. Therefore, many studies have also investigated other methods of conflict resolution, such as Deep Reinforcement Learning, which have shown positive results, but tend to be hard to explain due to their black-box nature. This paper investigates if it is possible to explain the behaviour of a Soft Actor-Critic model, trained for resolving vertical conflicts in a layered urban airspace, by interpreting the policy through a heat map of the selected actions. It was found that the model actively changes its policy depending on the degrees of freedom and has a tendency to adopt preventive behaviour on top of conflict resolution. This behaviour can be directly linked to a decrease in secondary conflicts when compared to analytical methods and can potentially be incorporated into these methods to improve them while maintaining explainability.@en

The traffic density of small aerial vehicles operating within urban environments is expected to increase significantly in the near future. This urban environment is highly constrained due to being limited to the low-altitude airspace directly above the existing road network. Multiple studies have addressed factors influencing the capacity of urban airspace. These have used simulations of aircraft, yet the empirical nature of these simulations limits their use beyond the specific conditions that have been tested. Analytical models would not have this limitation, but they are only developed for general airspace, while the emergent patterns in constrained urban airspace are different than in general, unconstrained conditions. For instance, queuing and local congestion are patterns that are unique to the heavily-constrained environment. Therefore, in this paper, we derive an analytical model for air traffic in a confined airspace to find the influencing factors for its capacity. By means of a simulation of aerial vehicles, we verify the analytical model and show a relationship between the mean flow rate and mean density in a two-dimensional orthogonal grid network airspace. Results show that the entire airspace can become unstable when the maximum capacity of just one intersection is reached. Furthermore, the maximum airspace density is found to be unaffected by cruise speed. The results demonstrate how the derived analytical model provides an effective tool to predict the effect of several design parameters on the capacity of constrained urban airspace. Moreover, this model can form the basis for further extensions, including the altitude dimension and non-orthogonal or non-four-way intersections.

@en

Unmanned Aircraft Systems (UAS) Traffic Management (UTM) is an active research subject as its proposed applications are increasing. UTM aims to enable a variety of UAS operations, including package delivery, infrastructure inspection, and emergency missions. That creates the need for extensive research on how to incorporate such traffic, as conventional methods and operations used in Air Traffic Management (ATM) are not suitable for constrained urban airspace. This paper proposes and compares several traffic capacity balancing methods developed for a UTM system designed to be used in highly dense, very low-level urban airspace. Three types of location-based dynamic traffic capacity management techniques are tested: street-based, grid-based, and cluster-based. The proposed systems are tested by simulating traffic within mixed (constrained and open) urban airspace based on the city of Vienna at five different traffic densities. Results show that using local, area-based clustering for capacity balancing within a UTM system improves safety, efficiency, and capacity metrics, especially when simulated or historical traffic data are used.

@en

The use of small aircraft for a wide range of missions in urban airspace is expected to increase in the future. In Europe, efforts have been invested into developing a unified system, called U-space, to manage aircraft in dense very-low-level urban airspace. The Metropolis II project aimed to research what degree of centralisation an air traffic management system should use in such airspace. The paper at hand is a follow-up, and investigates improvements that can be brought to the tactical conflict prevention, detection, and resolution module of such a system in order to harmonise these components with an organic high-density U-space environment. The proposed improvements are: the prioritisation of vertical conflict prevention in intersections, the use of intent in detecting and resolving conflicts, and the use of heading-based manoeuvres in open airspace. Results show that the use of intent information in the conflict detection process, as well as the implementation of suitable tactical prevention procedures, can greatly increase airspace safety. Furthermore, the experiments revealed that the effectiveness of conflict resolution algorithms is highly dependent on the airspace rules and structure. This reiterates the potential for increasing the safety and efficiency of operations within constrained airspace if the tactical separation modules are unified with the other components of air traffic management systems for U-space. @en

Current estimates show that the presence of unmanned aviation is likely to grow exponentially over the course of the next decades. Even with the more conservative estimates, these expected high traffic densities require a re-evaluation of the airspace structure to ensure safe and efficient operations. One structure that scored high on both the safety and efficiency metrics, as defined by the Metropolis project, is a layered airspace, where aircraft with an intended heading are assigned to a specific altitude layer. However, a problem arises once aircraft start to vertically traverse between these layers, leading to a large number of conflicts and intrusions. One way to potentially reduce the number of intrusions during these operations is by using conventional conflict resolution algorithms. These algorithms however have also been shown to lead to instabilities at higher traffic densities. As recent years have shown tremendous growth in the capabilities of Deep Reinforcement Learning, it is interesting to see how well these methods perform in the field of conflict resolution. This research investigates and compares the performance of multiple Soft Actor Critic models with the Modified Voltage Potential algorithm during vertical manoeuvres in a layered airspace. The final obtained performance of the trained models is comparable to that of the Modified Voltage Potential algorithm and in certain scenarios, the trained models even outperform the MVP algorithm. Overall, the results show that DRL can improve upon the current state of conflict resolution algorithms and provide new insight into the development of safe operations.@en

Deep Reinforcement Learning has seen more usage in the field of Air Traffic Control over the last couple of years. As the number of aircraft in a given sector of airspace is not constant, there is a need for methods to be invariant to the number of agents in the system. Often this is done by making a selection of the aircraft that will be included in the state, which introduces human biases. Another option that has been used is Recurrent Neural Networks to process the entire sequence of aircraft present. These methods however are sequence-dependent and can give different results depending on the order that the aircraft are given, which is undesirable. Methods that solely rely on attention mechanisms, such as transformers, allow sequential data to be processed in a sequence-invariant manner by using multi-head attention mechanisms. However, because traditional Transformers operate on individual tokens, this does not allow for relative state information to be encoded into the hidden state. This paper shows that by performing a transformation operation on the key and value tokens, it is possible to use Transformers on relative states, at the cost of a factor (N-1) additional attention computations, where N is the number of agents in the system. This adaptation allows relative state Transformers to obtain significantly higher performance than standard Transformers. The results also showed that using attention mechanisms to construct the initial observation vector out of a total of 20 agents results in similar, but slightly lower, performance to handcrafted observation vectors, without requiring manual selection of the important agents. Future research should investigate whether additional changes to the attention mechanisms and their training can result in higher performance.@en

Over the next decades, it is expected that the number of unmanned aerial vehicles (UAVs) operating in the airspace will grow rapidly. Both the FAA (Federal Aviation Administration) and the ICAO (International Civil Aviation Organisation) have already stated that aircraft operating autonomously or beyond their operators’ line of sight are required to have detect and avoid capabilities. At higher traffic densities these avoidance manoeuvres can, however, lead to instabilities within the airspace, causing emergent patterns that lead to knock-on effects that can harm the safety of the operations. It might be possible to formulate a cost function that encapsulates global safety, rather than individual safety, stimulating both safety and stability. One method that lends itself for optimizing such a cost function is cooperative Multi-Agent Reinforcement Learning (MARL). It has been demonstrated that MARL can be used for optimization in both competitive and cooperative (or even mixed) environments, however, when applied in a completely decentralized manner, stability issues often arise. It is therefore proposed to investigate the application of MARL for a well known centralized domain, ATC for manned aviation. This doctoral paper breaks down the proposed research project into 4 independent phases that individually contribute to the knowledge of applying MARL for ATC.@en

Metropolis II aims to provide insights in what is needed to enable high-density urban air operations. It does this by investigating the foundation for U-space U3/U4 services. The final goal is to provide a unified approach for strategic deconfliction, tactical deconfliction, and dynamic capacity management. Highly-dense operations in constrained urban airspace will likely require a degree of complexity that does not exist in modern-day air traffic management. The expected high traffic demand will require a shared use of the airspace instead of assigning exclusive use of blocks of the airspace to some flights. A unified approach for traffic management is needed because at high-densities, airspace design, flight planning, and separation management become increasingly interdependent. Metropolis II builds upon the results of the first Metropolis project. Three concepts with a varying degree of centralisation will be compared using simulations. (1) The centralised concept will take a global approach for separation management. (2) The decentralised concept aims to give the individual agents separation responsibility. (3) The hybrid concept tries to combine a centralised strategic planning agent with a robust tactical separation strategy.

@en

Future high traffic densities with drone operations are expected to exceed the number of aircraft that current air traffic control procedures can control simultaneously. Despite extensive research on geometric CR methods, at higher densities, their performance is hindered by the unpredictable emergent behaviour from surrounding aircraft. In response, research has shifted its attention to creating automated tools capable of generating conflict resolution (CR) actions adapted to the environment and not limited by man-made rules. Several works employing reinforcement learning (RL) methods for conflict resolution have been published recently. Although proving that they have potential, at their current development, the results of the practical implementation of these methods do not reach their expected theoretical performance. Consequently, RL applications cannot yet match the efficacy of geometric CR methods. Nevertheless, these applications can improve the set of rules that geometrical CR methods use to generate a CR manoeuvre. This work employs an RL method responsible for deciding the parameters that a geometric CR method uses to generate the CR manoeuvre for each conflict situation. The results show that this hybrid approach, combining the strengths of geometric CR and RL methods, reduces the total number of losses of minimum separation. Additionally, the large range of different optimal solutions found by the RL method shows that the rules of geometric CR method must be expanded, catering for different conflict geometries.@en

In order to enable the safe and efficient integration of Unmanned Aerial Vehicles into very low level airspace, current day research focuses on the development of new traffic services and procedures. One of these is the geovectoring protocol, which aims to reduce traffic complexity by setting limits on the allowed ground speed, course, and vertical speed. A geovector can be used to increase the capacity of an airspace by lowering the conflict rate. However, problems emerge when performing avoidance maneuvers in geovectored airspace, as the limits are ignored in this process. A powerful conflict resolution algorithm is the Modified Voltage Potential (MVP). This paper proposes an extension to the MVP algorithm, based on Velocity Obstacle theory. Making use of an alternative horizontal conflict resolution maneuver which respects the geovector, five resolution strategies are defined with different priority settings for the separate limits. The performance of these strategies is compared to pure MVP on geovector, safety, and stability measures, making use of fast-time simulations in a corridor airspace. All geovector resolution strategies show improvements on the ability to perform conflict resolution maneuvers within the geovector limits, while having marginal effects on the overall airspace safety level. It is recommended to further investigate the performance of the geovector resolution strategies for other types of airspace, to verify whether the observed reduction in conflict rate from the geovectors can be reinforced by the resolution strategies.@en

Current predictions on future drone operations estimate that traffic density orders of magnitude will be higher than any observed in manned aviation. Such densities redirect the focus towards elements that can decrease conflict rate and severity, with special emphasis on airspace structures, an element that has been overlooked within distributed environments in the past. This work delves into the impacts of different airspace structures in multiple traffic scenarios, and how appropriate structures can increase the safety of future drone operations in urban airspace. First, reinforcement learning was used to define optimal heading range distributions with a layered airspace concept. Second, transition layers were reserved to facilitate the vertical deviation between cruising layers and conflict avoidance. The effects of traffic density, non-linear routes, and vertical deviation between layers were tested in an open-source airspace simulation platform. Results show that optimal structuring catered to the current traffic scenario improves airspace usage by correctly segmenting aircraft according to their flight routes. The number of conflicts and losses of minimum separation was reduced versus using a single, uniform airspace structure for all traffic scenarios, thus enabling higher airspace capacity.

@en

Future operations involving drones are expected to result in traffic densities that are orders of magnitude higher than any observed in manned aviation. Current geometric conflict resolution (CR) methods have proven to be very efficient at relatively moderate densities. However, at higher densities, performance is hindered by the unpredictable emergent behaviour from neighbouring aircraft. Reinforcement learning (RL) techniques are often capable of identifying emerging patterns through training in the environment. Although some work has started introducing RL to resolve conflicts and ensure separation between aircraft, it is not clear how to employ these methods with a higher number of aircraft, and whether these can compare to or even surpass the performance of current CR geometric methods. In this work, we employ an RL method for distributed conflict resolution; the method is completely responsible for guaranteeing minimum separation of all aircraft during operation. Two different action formulations are tested: (1) where the RL method controls heading, and speed variation; (2) where the RL method controls heading, speed, and altitude variation. The final safety values are directly compared to a state-of-the-art distributed CR algorithm, the Modified Voltage Potential (MVP) method. Although, overall, the RL method is not as efficient as MVP in reducing the total number of losses of minimum separation, its actions help identify favourable patterns to avoid conflicts. The RL method has a more preventive behaviour, defending in advance against nearby neighbouring aircraft not yet in conflict, and head-on conflicts while intruders are still far away.

@en

Air traffic demand has increased at an unprecedented rate in the last decade (albeit interrupted by the COVID pandemic), but capacity has not increased at the same rate. Higher levels of automation and the implementation of decision-support tools for air traffic controllers could help increase capacity and catch up with demand. The air traffic control problem can be effectively modelled as a Markov game, where a team of aircraft (the agents) interact in the airspace (the environment) and cooperatively take resolution actions to achieve a common goal: safe separation in the most efficient way. As in any Markov game, the optimal policy for the team could be learnt through trial and error in a simulated environment using reinforcement learning algorithms. In this paper, we use the soft actor-critic algorithm to unravel the optimal air traffic control policy. Unlike some previous works, we propose a global (i.e., shared) reward that encourages cooperative behaviour. Furthermore, we propose a versatile policy model capable of performing heading, speed, and/or altitude resolution actions. We also demonstrate that the policy is robust and can maintain safe separation even in the presence of uncertainty regarding aircraft position, delays in implementing resolution actions, and wind. The findings of this paper also suggest that there is still significant room for improvement when controlling three degrees of freedom at the same time.@en