Incremental Dual Heuristic Programming (IDHP) is a successor to the Dual Heuristic Programming (DHP) algorithm that uses an online identified incremental system model, this algorithm showed promising flight control performance and tolerance of faults in simulation experiments. This paper studies the potential for extending IDHP through augmenting the computation of agent updates and returns, more specifically, by using eligibility trace updates and multi-step temporal difference error. This results in the IDHP(𝜆), MIDHP, and MIDHP(𝜆) algorithms, which are compared against IDHP in several simulated flight control scenarios with faults introduced mid-flight. The results demonstrate that the proposed algorithms have improved flight control performance and fault tolerance in terms of tracking errors when controlling a nominal aircraft and an aircraft with faults introduced, with the most improvement observed in MIDHP(𝜆)

Exploring planetary bodies using robot swarms can potentially increase the value of the exploration missions; enabling the execution of novel measurements and explorations previously deemed impractical or unattainable. Despite its potential, the technology readiness level of planetary swarms is not very mature. This work uses multi-agent reinforcement learning to find control policies that allow swarms to autonomously explore unknown areas in a decentralized manner, contributing towards the technology readiness of the field. A multi-agent proximal policy optimization (MAPPO) algorithm is proposed for this end, where the policy uses LIDAR perception information, and the input of the value function contains local and global environment information. The algorithm finds control policies that achieve cooperation behaviors and generalize to unseen swarm sizes and environments learning with simple, sparse reward functions. Moreover, different types of reward functions, value inputs, and environment configurations are investigated. Compared with the state-of-the-art in the field, MAPPO can learn with a larger number of agents, more complicated environments, and using sparse rewards instead of dense ones.

Reinforcement Learning applied to flight control has shown to have several benefits over classical, linear flight controllers, as it eliminates the need for gain scheduling and it could provide fault-tolerance. The application to civil aviation in practice, however, is non-existent as there are multiple safety concerns. This research demonstrates the evaluation of longitudinal Handling Qualities of the Soft Actor-Critic Deep Reinforcement Learning framework with the aim to translate the unpredictable black box of Reinforcement Learning into classical flight control terminology. The framework is applied to a pitch rate command system of a jet aircraft and shows robustness to off-nominal flight conditions, center of gravity shifts and biased sensor noise. Accurate tracking performance is achieved, while adhering to Level 1 longitudinal Handling Qualities for all conditions.

The Flying-V is a tailless, V-shaped flying-wing type aircraft that promises to offer significant increases in aerodynamic efficiency. Due to its configuration, the Flying-V faces some control and stability related issues. These include limited control authority, pitch break tendencies and non-ideal handling qualities. To enhance the handling qualities of the Flying-V, Incremental Nonlinear Dynamic Inversion (INDI)-based flight control systems have been proposed. INDI, a sensor-based alternative to conventional Nonlinear Dynamic Inversion (NDI), is rooted in the principle of feedback linearization. Unlike NDI, INDI does not depend heavily on accurate on-board models (OBM), thereby offering increased robustness to aerodynamic uncertainties. However, singular perturbations—such as time delays, aeroelastic effects, and additional unmodeled or unknown dynamics—have been identified as challenges for INDI-based control laws. Various strategies have been explored to improve the overall robustness of INDI-based flight control systems, including outer-loop tuning and inversion loop augmentation strategies. In this research a multi-loop µ-optimal approach for designing robust inversion-based flight control laws is explored for the design of an explicit model-following pitch-rate control system for a short-period approximation of the Flying-V’s longitudinal dynamics. The design problem takes into account both regular and singular perturbations. To assess the robust stability and performance of the proposed control systems, a structured singular value analysis was performed. It was concluded that a multi-loop synthesis approach is capable of achieving better robust stability and performance levels when compared to either strictly inner-loop or outer-loop synthesis. As such, it can be concluded that multi-loop synthesis approaches are best capable of leveraging the robustness functionalities of multi-loop inversion-based control systems.

The Flying-V aircraft could revolutionize commercial aviation, boasting a potential 25% increase in aerodynamic efficiency. Due to inherent design limitations regarding static stability, the need for a proper Flight Control System (FCS) is essential for the development of the aircraft. The concept of Hybrid Incremental Nonlinear Dynamic Inversion (INDI) was introduced to mitigate the insufficient stability margin encountered in existing sensor-based INDI systems due to sensor time delays, to achieve Level 1 Handling Qualities (HQ). Furthermore, the research introduces an exponential potential function-based command limiting Flight Envelope Protection (FEP) to enhance safety compared to the currently implemented linear-based FEP. The study compares and evaluates the effectiveness of the updated system under various flight conditions and parametric uncertainties. Results show improved stability margins and a safer FEP. However, additional research is required into actuator saturation and control allocation issues during the approach condition and to enhance fault tolerance.

In the rapidly evolving aviation sector, the quest for safer and more efficient flight operations has historically relied on traditional Automatic Flight Control Systems (AFCS) based on high-fidelity models. However, such models not only incur high development costs but also struggle to adapt to new, complex aircraft designs and unexpected operational conditions. As an alternative, deep Reinforcement Learning (RL) has emerged as a promising solution for model-free, adaptive flight control. Yet, RL-based approaches pose significant challenges in terms of sample efficiency and safety assurance. Addressing these gaps, this paper introduces Returns Uncertainty-Navigated Distributional Soft Actor-Critic (RUN-DSAC). Designed to enhance the learning efficiency, adaptability, and safety of flight control systems, RUN-DSAC leverages the rich uncertainty information inherent in the returns distribution to refine the decision-making process. When applied to the attitude tracking task on a high-fidelity non-linear fixed-wing aircraft model, RUN-DSAC demonstrates superior performance in learning efficiency, adaptability to varied and unforeseen flight scenarios, and robustness in fault tolerance that outperforms the current state-of-the-art SAC and DSAC algorithms.

The critical challenge for employing autonomous control systems in aircraft is ensuring robustness and safety. This study introduces an intelligent and fault-tolerant controller that merges two Reinforcement Learning (RL) algorithms in a hybrid approach: the Distributional Soft Actor-Critic (DSAC) and the Incremental Dual Heuristic Programming (IDHP). The integration combines the strengths of DSAC in learning a robust control strategy and IDHP in allowing real-time control adaption. Compared to earlier controllers, such as a hybrid using the Soft Actor-Critic (SAC) algorithm and strictly offline DSAC and SAC, our hybrid demonstrates enhanced robustness against changing flight conditions and in the face of sensor noise and bias. During fault tolerance tests, it maintains superior control even when the effectiveness of the aircraft’s ailerons and elevators is compromised. By demonstrating the potential of RL-based controllers to provide robustness and fault tolerance, this research advances the feasibility of safe and autonomous flight control operations.

Power output during flight operation of multi-megawatt airborne wind energy systems is substantially affected by the mass of the airborne subsystem, resulting in power fluctuations. In this paper, an approach to control the tether force using the airborne subsystem is presented that improves the quality of the power output. This kite tether force control concept is implemented on the 3DOF dynamic simulation of the MegAWES reference model. First, the winch of MegAWES is resized because an analysis of winch inertia and radius shows its effect on power output and tether force overshoot. Second, the power consuming sections during the traction phase are eliminated by using a feedforward winch controller. Finally, the peak power is substantially reduced by implementing the kite tether force controller which uses a measurement of the tether force, angle of attack, and airspeed to keep the tether force constant when the system is at its power limit. This reduces the range between minimum and maximum power output by 75%.

With the alignment of the planets in the 2030s, a perfect opportunity is presented to explore the outer reaches of the Solar System. With this, studying Uranus would become possible. With it being one of the least studied planets in the Solar System, the demand for accurate scientific data is high. This is where Ouranos comes in. This project has the specific purpose to design a scientific mission to Uranus, where measurements will be taken of its atmosphere. The summary below highlights the important parts and results of this report.

This research presents a comprehensive modeling approach for the flight dynamics of a hybrid compound helicopter, employing classical mechanics methods. The derived non-linear mathematical model encompasses the individual components of the aircraft, including the rotor, propellers, wings, fuselage, and empennage, which are then integrated into a unified dynamics framework through the final Equations of Motion. Notably, the model incorporates a conventional first-order steady flapping motion and quasi-dynamic inflow modeling for both the rotor and propellers. The resulting model represents a complete 9 Degree-of-Freedom system, augmented with 6 mechanical ones and an additional 3 for the inflows of the rotor and the two propellers. As an over-actuated coupled system, it offers 7 available controls; rotor collective pitch, rotor longitudinal and lateral cyclic pitch, port-side and starboard-side propeller pitch, elevator deflection, and rudder deflection. The primary objective of this study is to identify operating trim points during forward flight, ranging from hover to the observed maximum airspeed of 255 knots. Achieving trim through numerical optimization involves a control allocation process utilizing a mapping function that prioritizes the auxiliary propulsion. The research findings demonstrate the successful implementation of a slowed-rotor strategy at high speeds, effectively mitigating compressibility effects, and a linearized mathematical system is derived, providing essential insights for future stability analysis and control system design. These outcomes contribute valuable information toward the advancement of hybrid compound helicopter technology.

This paper investigates the performance of an autonomous navigation system to navigate a spacecraft in the proximity of a binary asteroid system using optical and laser ranging measurements. The knowledge about the binary asteroid is limited to its orbital parameters and ellipsoid shape models. The accelerometer bias random walk is included in the estimation process. Over a four-hour landing maneuver starting from 6770 m altitude and ending at 550 m, the mean position estimation uncertainty is 41.6 m (3σ). It is shown that the navigation accuracy is sensitive to the Sun phase angle, the irregularity of the asteroid shape, and the goodness of fit of the ellipsoid shape model. The paper demonstrates that the navigation system is robust to large errors in the initialization of the extended Kalman filter state. The impact of image distortion and two types of image noise on the navigation performance are investigated.

Control of a helicopter with a deployed dipping sound navigation and ranging (SONAR) is no trivial task due to the complex dynamics of the suspension cable. The cable can change shape, which influences the effect of water, wind and the motion of the helicopter itself. To control such a system, a control method is needed that can cope with these complex dynamics.
In this paper, an investigation is performed on how to model and control a helicopter and a dipping SONAR through the suspension cable using incremental nonlinear dynamic inversion (INDI) as well as a comparison between the use of INDI and linear proportional integral derivative (PID) control. The controller uses only the cable states at the attachment point of the cable to the helicopter in order to determine the control inputs to the helicopter. This means that only these measurements are needed and no complex model of the cable is required for control. This reduces model dependency of the controller.
The control system is simulated in different wind conditions in order to analyse its performance in turbulent conditions, where the cable changes shape constantly. It was found that, although INDI performs better than PID control at low wind speeds, an INDI position hold controller appeared to perform better than an INDI cable controller at high wind speeds.

Recent research in bio-inspired artificial intelligence potentially provides solutions to the challenging problem of designing fault-tolerant and robust flight control systems. The current work proposes SERL, a novel Safety-informed Evolutionary Reinforcement Learning algorithm, which combines Deep Reinforcement Learning (DRL) and neuro-evolutionary mechanisms. This hybrid method optimises a diverse population of non-linear control policies through both evolutionary mechanisms and gradient-based updates. We apply it to solve the attitude tracking task on a high-fidelity non-linear fixed-wing aircraft model. Compared to a state-of-the-art DRL solution, SERL achieves better tracking performance in nine out of ten cases, remaining robust against faults, changes in initial conditions and external disturbances. Furthermore, the work shows how evolutionary mechanisms can balance performance with the smoothness of control actions, a feature relevant for bridging the gap between simulation and deployment on real flight hardware.

Even though Deep Reinforcement Learning (DRL) techniques have proven their ability to solve highly complex control tasks, the opaqueness and inexplicability associated with these solutions many times stops them from being applied to real flight control applications. In this research, reward decomposition explanations are used to tackle this issue and augment DRL end-user explainability. A reward decomposition-based DRL controller is deployed in a longitudinal state-space model of the Cessna Citation 500 aircraft, and it is assessed on two attitude flight control tasks. Furthermore, a new explanation type called Dominant Reward eXplanations (DRX) is presented, which allows users to obtain more global insights than the ones generated by Reward Difference eXplanations (RDX). Results show that the explanations produced lead to straightforward and intuitive insights about the controller’s behaviour, capable of improving end-user explainability. Moreover, a small analysis seems to indicate that the decomposed method has similar performance to the one obtained without reward decomposition, however, training time increases considerably. To the author’s best knowledge, this is the first application of reward decomposition explanations to the flight control domain.

Recent advancements in fault-tolerant flight control have involved model-free offline and online Reinforcement Learning algorithms in order to provide robust and adaptive control to autonomous systems. Inspired by recent work on Incremental Dual Heuristic Programming (IDHP) and Soft Actor-Critic (SAC), this research proposes a hybrid SAC-IDHP framework aiming to combine adaptive online learning of IDHP with the high complexity generalization power of SAC in a fully coupled system. Using principles from transfer learning, the architecture of the SAC-IDHP hybrid policy is designed as alternating pre-trained SAC layers and online learning identity initialized IDHP layers with the SAC layers frozen during online learning. This hybrid framework is implemented into the inner loop of a cascaded altitude controller for a high-fidelity, six-degree-of-freedom model of the Cessna Citation II PH-LAB research aircraft. Multiple altitude tracking tasks with coordinated turns are simulated to compare the tracking performance to SAC-only in several failure modes. Compared to SAC-only, the SAC-IDHP hybrid demonstrates an improvement in tracking performance of 0.74%, 5.46% and 0.82% in normalized Mean Absolute Error for nominal case, longitudinal and lateral failure cases respectively. Additionally, random online policy initialization is eliminated due to identity initialization of the hybrid policy, resulting in an argument for increased safety.

Recent research on the Flying V - a flying-wing long-range passenger aircraft - shows that its airframe design is 25% more aerodynamically efficient than a conventional tube-and-wing airframe. The Flying V is therefore a promising contribution towards reduction in climate impact of long-haul flights. However, some design aspects of the Flying V still remain to be investigated, one of which is automatic flight control. Due to the unconventional airframe shape of the Flying V, aerodynamic modelling cannot rely on validated aerodynamic-modelling tools and the accuracy of the aerodynamic model is uncertain. Therefore, this contribution investigates how an automatic flight controller that is robust to aerodynamic-model uncertainty can be developed, by utilising Twin-Delayed Deep Deterministic Policy Gradient (TD3) - a recent deep-reinforcement-learning algorithm. The results show that an offline-trained single-loop altitude controller that is fully based on TD3 can track a given altitude-reference signal and is robust to aerodynamic-model uncertainty of more than 25%.

Reinforcement Learning is being increasingly applied to flight control tasks, with the objective of developing truly autonomous flying vehicles able to traverse highly variable environments and adapt to unknown situations or possible failures. However, the development of these increasingly complex models and algorithms further reduces our understanding of their inner workings. This can affect the safety and reliability of the algorithms, as it is difficult or even impossible to determine which are their failure characteristics and how they will react in situations never tested before. It is possible to remedy this lack of understanding through the development of eXplainable Artificial Intelligence and eXplainable Reinforcement Learning methods like SHapley Additive Explanations. In this thesis, this tool is used to analyze the strategy learnt by an Actor-Critic Incremental Dual Heuristic Programming controller architecture when presented with a series of pitch rate and roll rate tracking tasks in a variety of non-linear flying conditions, which include flying close to the stall regime, experiencing non-linear reference signals, reaching the control surface deflection limits, and flying with large sideslip angles. This same controller architecture has been previously explored with the same analysis tool but limited to the nominal linear flight regime, and it was observed that the controller learnt linear control laws, even though its Artificial Neural Networks should be able to approximate any function. This thesis shows that, even in the non-linear flight regime, it is still more optimal for this controller architecture to learn quasi-linear control laws, although it seems to continuously modify the linear slopes as if it was an extreme case of the gain scheduling technique. Additionally, a more complex Reinforcement Learning architecture based on the Soft Actor-Critic algorithm is also explored with the same analysis tool, to demonstrate its usability in the presence of non-linear control laws and to improve our understanding of this offline state-of-the-art algorithm.