This thesis investigates the application of alternating maximisation for active wake control in wind farms, focusing on both numerical static yaw optimisation and multi-agent deep reinforcement learning for dynamic yaw control. As the size and number of offshore wind farms continue to grow, effectively mitigating wake effects ---~where the airflow behind a turbine is disturbed, reducing the efficiency of downstream turbines~--- becomes increasingly important to maximise power output and ensure economic profitability.

Current numerical strategies are either sub-optimal, as they do not leverage the shape of the optimisation surface well, or expensive to optimise due to the computational demands of high-fidelity simulations. Similarly, current reinforcement learning approaches are too sample inefficient on contemporary wind farms of 40+ turbines to learn high-quality policies and likely suffer from credit assignment problems, other agents performing exploration, and relative over-generalisation. Alternating maximisation emerges as a promising candidate to address these challenges by providing an efficient and potentially more effective optimisation method.

In static yaw optimisation, the research analysed the effects of wind direction, wind speed, turbine layout, and yaw misalignment on the optimisation surface. Key findings indicate that Nash equilibria are transient and significantly influenced by these factors.

The sampling-based approach to alternating maximisation outperformed other methods, proving robust against variations in initial yaw configurations and optimisation orders. This approach required fewer samples and appears suitable for practical applications.

For dynamic yaw control, alternating maximisation with policy gradients demonstrated greater sample efficiency compared to distributed policy gradient methods. However, convergence to sub-optimal policies was common due to limited exploration capabilities. An oscillating exploration strategy enabled effective exploration of yaw angles, leading to high-quality policies.

The study primarily focused on farm power output, neglecting structural loading and leading-edge erosion. Future research should integrate these factors and explore dynamic yaw control under time-varying wind conditions.

Overall, alternating maximisation shows potential in enhancing wind farm performance through efficient optimisation strategies. Future work should refine these methods and address identified limitations to maximise practical applicability.

The increasing demand for electricity has lead to demand for more efficient energy production. One promising option is wind power, which currently provides an estimated 7.8% of the world’s energy production. One of the problems with wind energy is that a small percentage of the energy is lost due to the wake effect. The wake of a wind turbine is an area of low wind speed and high turbulence which is caused by the spinning of the turbine. This wake effect can mitigated by active wake control, which is a process by which the wake from a turbine is redirected away from downwind turbines, by changing the yaw of the turbine head. Calculating a policy for doing this is computationally expensive to do using numerical optimisation. Therefore, multi agent reinforcement learning is proposed to learn a policy which performs active wake control.
The proposed approach makes use of the popular reinforcement learning algorithm REINFORCE, and extends it using a variety of methods. First, a simplified version of the problem is treated, wherein the wind direction is fixed. Then the problem is made more realistic by introducing changing wind directions. The first extension of REINFORCE that is treated is difference rewards, a reward shaping strategy which seeks to solve the credit assignment problem, thereby improving cooperation between turbines. The second method uses training regimes, which train different agents at different times to stabilise the environment as much as possible. Next, role-based reinforcement learning is used to conteract the complexity of the problem by allowing each agent to specialise for a certain role. Finally, since roles cannot be manually determined for larger farms, influence-based abstraction is used to enable agents to learn the roles themselves, by abstracting spacial information and presenting it to the agent as an observation.
The results demonstrate that multi agent reinforcement learning can be used to perform active wake control in wind farms. Furthermore, the extensions proposed are shown to improve learning, and lead to greater energy output. While multi agent reinforcement learning is shown to be a promising way to tackle active wake control in wind farms, research is needed to improve the stability of the learned policies.

The wake effect which is turbulence behind a wind turbine created when it extracts energy negatively impacts the power output of the downstream turbines. Active Wake Control can mitigate this effect, by rotating some turbines away from the wind. Previous research applied single agent reinforcement learning to apply Active Wake Control, show- ing good results for small-scale layouts, that don’t scale for larger, practical wind farms.
To that extent, this study focuses on the application of mean-field multi-agent reinforcement learning to Active Wake Control, under constant wind conditions. This algorithm limits the computations to a limited set of neighbouring turbines, reducing their complexities. To build the answer to this question I will also study:
1. how to model the rewards to solve the lazy- agent problem, leveraging the nature of the Active Wake Control
2. how the view of the agent changes the results
3. how does it compare to a single-agent reinforcement learning algorithm, TD3
The experiments were done using the Floris Wake Simulator, with each turbine sharing the same agent, placed in tunnel layouts at real-life distances (6-7 rotor diameters), under constant wind conditions.
Results show that with the proper configuration of rewards and view space within wind tunnels, the mean-field algorithm finds near optimal configurations for Active Wake Control, within a small number of episodes. This shows a promising start for the application of mean-field multi-agent algorithms for the Active Wake Control problem, and provides insight into how to model the rewards, which might be applicable for the whole class of algorithms.

When multiple wind turbines are positioned close to one another, such as in a wind farm, wind turbines located downwind of other turbines are not 100% efficient due to wakes, negatively affecting the total power output of the wind farm. A way to mitigate the loss of power is to steer the wake away from the next turbine, which lowers the current turbine's power output but increases the turbine's total power output. As the number of wind turbines increases, how complex it is to calculate the optimal steering increases exponentially. Reinforcement learning techniques have been a promising candidate to solve this problem. However, single-agent techniques are still very computationally expensive when the number of turbines is the same as an average wind farm. Therefore, this paper aims to see how the QMIX algorithm, a multi-agent reinforcement learning technique, can be efficiently applied to the problem of active wake control. QMIX will be compared to the FLORIS model and a single agent deep reinforcement learning technique TD3 to see if it achieves a higher average reward and converges faster. Finally, QMIX is tested on a larger wind farm to see if it achieves any results in a reasonable amount of time, showing that multi-agent reinforcement learning techniques are more suitable to the problem. This paper shows QMIX has the potential to outperform TD3; although FLORIS is better for smaller wind farms but more research has to be done in applying QMIX to the problem.

The close proximity of wind turbines to one another in a wind farm can lead to inefficiency in terms of power production due to wake effects. One technique to mitigate the losses is to veer from their individual optimal direction. As such, the wakes can be steered away from downstream turbines in order to increase the overall power output. Multi-Agent Reinforcement Learning (MARL) models the interactions between wind turbines and determines an optimal control strategy through agents that learn the collective consequences of their actions. To
analyse the benefit of multi-agent cooperation and centralised critic evaluation, I investigated the effect of Counterfactual Multi-Agent Policy Gradients (COMA) on Active Wake Control. Ultimately, experiments on wind farms of three and sixteen turbines indicate that the algorithm performs moderately, yet worse than single-agent Reinforcement Learning. In addition, high computation costs hinder its application on real-life environments.

Wind energy, generated by windfarms, is playing an increasingly critical role in meeting current and future energy demands. windfarms, however, face a challenge due to the inherent flaw of wake-induced power losses when turbines are located in close proximity. Wakes, characterized by regions of turbulence and lower wind speed, are created as air passes through the rotors, reducing the efficiency of downstream turbines. Wake losses can be reduced by yawing upstream turbines to steer the wake away from downstream turbines. While there are power losses associated with turning turbines off-wind, the gains in the subsequent turbines can outweigh these losses. Yawing turbines to increase overall power output is known as Active Wake Control, and the literature shows that single-agent reinforcement learning algorithms can be used to learn such control policies. However, these approaches are limited to a small number of turbines and do not scale well to larger windfarms. Multi agent reinforcement learning algorithms do scale to larger windfarms and this paper investigates the application of the DGN algorithm for windfarm active wake control. DGN is a fully cooperative algorithm that utilizes a graph representation of agents to encourage collaboration among neighboring turbines. The use of DGN is particularly interesting because windfarms naturally have a topological structure, and depending on the modeling choices, graphs can capture a significant amount of information. This paper demonstrates that DGN can learn useful policies for wake control. Although it does not outperform the DQN single-agent algorithm on small windfarms, its advantage becomes apparent in larger windfarms, where its performance remains consistent while DQN’s performance deteriorates.

In wind farms wind turbines are often placed close to each other. Each turbine generates a turbulent wake field, this field negatively affects subsequent turbines. This can cost more than 12% efficiency. To decrease this loss we can steer the turbines away from the wind direction, this will decrease the individual turbine power output, but can increase the total power output of the farm. As the size of the farm increases the number of possible actions increase exponentially. Due to this a numerical solution is not feasible. A reinforcement learning technique has been proven useful in the past, but a standard single agent implementation is still very computationally expensive. We evaluate the effectiveness of MADDPG on the active wake control problem. MADDPG is a multi agent reinforcement learning algorithm. MADDPG will be compared to the numerical solver FLORIS and to the already implemented and proven TD3 (which is a variation on a single agent DDPG algorithm). We compare the eventual output power of the algorithms with MADDPG. From the results we can see that MADDPG does improve on the learning performance of TD3, but since MADDPG needs to manage more neural networks the overhead is larger. MADDPG reaches an optimum solution in less training steps, but these steps take significantly more time.

Automated asset trading is a crucial method used by financial entities such as investment firms or hedge funds. It allows them to allocate their capital in order to maximize their rate of returns. In scientific literature, there are multiple models suggested to solve this problem. However, these models either lack the complexity to understand the market or scalability for the market in general. On the other hand, deep reinforcement learning is a great framework that can solve these problem. In this study we aim to understand the performance of model-free deep reinforcement learning algorithms in terms of training speed, financial performance and generalizability by training and comparing them on a smaller representative market. Proximal Policy Approximation (PPO) and Twin Delayed Deep Deterministic Policy Gradient (TD3) were used as a representatives of policy approximation and QLearning algorithms respectively. Our study have found that while proximal policy algorithms offer higher speed due to smaller training data they use at each timestep, Q-Learning algorithms offer a better general performance in terms of stability and generalizability. With respect to financial performance on training stocks, this study did not find a statistically important difference in performances.

One of the most challenging types of environments for a Deep Reinforcement Learning agent to learn in are those with sparse reward functions. There exist algorithms that are designed to perform well in settings with sparse rewards, but they are often applied to continuous state-action spaces, since economically relevant problems like robotic control and stock trading fall under this category. This means the continuous version overshadows the discrete state-action version of the sparse reward problem. Furthermore, research that focuses on sparse rewards is lacking in comparisons of algorithms dedicated to performing in this type of setting with other state-of-the-art Deep Reinforcement Learning algorithms. We devise an experimental setup to test a selection of algorithms from three state-of-the-art Deep Reinforcement Learning approaches; Hindsight Experience Replay, Maximum Entropy Reinforcement Learning and Distributional Reinforcement Learning. We show that as the cardinality of the state spaces in sparse reward settings increase, Hindsight Experience Replay approaches are superior in sample efficiency compared to the other two approaches studied.

The current state-of-the-art solutions for playing Chess, are created using deep reinforcement learning. AlphaZero, the current world champion, uses ’policy networks’ and ’value network’ for selecting moves and evaluating positions respectively. However, the training of these networks are done using reinforcement learning from games of selfplay. There are many factors which determine the learning speed of reinforcement learning algorithms, where the size of the search space is a main one. In this research, we have tried to see the effect of the size of the search space on the time it takes the reinforcement learning agent to learn.