Autonomous aircraft learns to judge its own abilities before flying missions

The future of aviation is quietly shifting the human out of the cockpit and into a chair. Urban air taxis, airborne package couriers, and military surveillance drones are all moving toward a model in which a single supervisor, often far from the aircraft, oversees one or more autonomous platforms. But this new paradigm of remote mission supervision carries a hidden danger: the operator may have only a vague idea of what the aircraft can actually do. A team of researchers at the University of Colorado Boulder and Draper has now built a learning-based autonomous uncrewed aircraft system (UAS) that can quantify its own abilities and report them to a human partner before a mission begins. Their work, published in Frontiers in Aerospace Engineering, demonstrates a computational method called generalized outcome assessment (GOA) that lets an aircraft essentially answer the question every supervisor wants to ask: can you actually do this job?

The research targets a use case known as small-unit intelligence, surveillance, and reconnaissance (ISR). Small ISR platforms are person-portable aircraft that can be thrown or catapulted into the air by a fire crew searching for a blaze or a squad of soldiers scouting beyond the next hill. These platforms have basic onboard autonomy for path planning and waypoint following, but they are constrained by size, weight, and power, and they are vulnerable to inclement weather. Crucially, their operators often receive far less training than pilots of larger military drones. Soldiers operating the MQ-7 RAVEN, for example, require only 10 days of training. That limited preparation means supervisors may have little understanding of how the aircraft behaves in off-nominal situations such as poor weather, and any misunderstanding can be amplified by the high-stress environments in which these systems are deployed.

The core idea behind the new work is machine self-confidence, a concept the researchers describe as an expression of machine self-trust, akin to the self-confidence a human subordinate expresses to a supervisor who has delegated a task. The team built on the factorized machine self-confidence (FaMSeC) framework, which breaks an autonomous agent’s introspective self-assessment into interrelated meta-factors. For agents reasoning under standard Markov decision process models, three key factors can be evaluated: outcome assessment, which asks whether possible events and rewards lead to a desirable landscape of outcomes under uncertainty; solver quality, which asks whether the approximations used to solve decision-making problems are appropriate; and model validity, which asks whether the agent’s learned models and training data are sufficient for the real world. Scores for each factor map onto bounded scales, where the lower bound signals complete lack of confidence and the upper bound signals complete confidence.

Applying FaMSeC to learning-based agents is not straightforward. Classical outcome assessment formulations assume tractable closed-form state transition probability distributions, fixed policy functions, and static reward functions. In reinforcement learning applications with continuous state spaces and online model predictive control, all three assumptions break down. The researchers also identified weaknesses in the classical outcome assessment metric: cumulative rewards contain no information about specific intermediary outcomes, reward values are not necessarily proportional to user desirability, and the user-defined margin of success implicitly requires the user to understand the reward function. To address these gaps, the team proposed generalized outcome assessment, which quantifies confidence that a specified margin of success can be achieved based on a set of potential outcomes rather than abstract reward values.

GOA works by running simulation traces and collecting raw outcome statistics of interest, such as the number of waypoints reached or the battery state at mission end. These raw outcomes are first broken down into unranked equivalence classes, then ranked by order of user desirability and mapped into a new domain. For example, battery levels could be binned into ranges, with higher bins preferred. Given a histogram of simulated outcomes, the method defines a minimum expectation of performance directly related to mission outcomes, which the researchers argue is far more interpretable to users than reward-based thresholds. If an operator wants to know whether the aircraft can complete a mission with at least 10 to 20 percent battery remaining, they simply set that bin as their minimum acceptable level. The ratio of upper to lower partial moments of the outcome distribution relative to that threshold is then passed through a logistic function, producing a GOA score between -1 and 1 that is ultimately mapped into semantic confidence statements for the user.

To demonstrate the approach, the team developed a simulated multi-target ISR mission in the Gazebo robotics simulator, modeling a lightweight fixed-wing platform similar to the MQ-7 RAVEN. The simulated aircraft, based on the Techpod UAS from the RotorS package, has a wingspan of 2.6 meters, a body length of 1.1 meters, and a mass of 2.6 kilograms. Its six-dimensional action space comprises control surface deflections of the ailerons, elevator, and rudder, plus thrust. A linear battery model drains the battery proportionally to commanded thrust, tuned so the aircraft has roughly 60 seconds of flight time at maximum thrust. Custom Gazebo plugins modeled the platform dynamics, battery, and environmental winds, with all communication handled through the Robot Operating System.

The heart of the learning system is a probabilistic world model trained via model-based reinforcement learning. The complete state space is 18-dimensional, capturing translational and rotational positions and velocities, wind velocity, battery level, external air temperature, and payload mass. The researchers trained a recurrent neural network with a GRU cell on 40,000 trajectories collected by executing random actions in the simulation, with data sampled at 1 Hz over 60-second sequences. The network predicts ten time-steps ahead, with the total loss computed as the sum of mean absolute errors at each step. To handle the challenges of absolute positions and Euler angles, the team formulated inputs and outputs in what they call the model frame, where gravity is constant, predicting changes in pose rather than absolute vehicle pose and then integrating the network output. During planning, the trained world model runs within a model predictive control framework alongside a filtering and reward-weighted refinement planner, re-planning at every time step so only the first action of the best action sequence is applied to the environment.

The experimental design tasked the autonomous aircraft with visiting six fixed targets dispersed throughout a mission area within a fixed 60-second mission, under two wind conditions: calm winds representing a nominal environment, and adverse winds representing an off-nominal one. The reward function combined three elements: a target reward for capturing a target by flying within a 25-meter radius, a penalty for crashing into the ground plane, and a reward for conserving battery life. The team decomposed the high-level mission goal into three outcomes of interest to a potential operator: battery-level conservation, time to the first visit of any target, and total targets visited. For each wind condition, GOA sampled 25 rollouts from the simulation, producing sets of 60-second trajectory traces that were then analyzed with respect to each outcome. All experiments ran on a Dell Precision laptop with an Intel i7 processor, 16 GB of RAM, and an Nvidia RTX A3000 GPU, with the software running in Docker containers.

The results reveal how wind conditions reshape both performance and self-assessed confidence. In calm winds, trajectories followed smooth, straight paths from one target to the next, and the aircraft almost always visited three targets. In adverse winds, paths became meandering and random, with predictions indicating the aircraft could achieve as many as four visits or as few as none. Across the battery outcome, GOA estimated very good confidence of completing the mission with at least a modest reserve in both conditions, but strictly higher confidence in calm winds for ending battery levels in the middle range, since the aircraft must expend more thrust countering stronger winds. For target timing, both conditions tracked similar curves, with very bad confidence of reaching the first target before roughly 20 seconds and very good confidence for all later times. Notably, the GOA curve in adverse winds was shallower, indicating the method loses confidence faster when outcome distributions carry greater risk. Interestingly, GOA showed a small amount of confidence that the aircraft could visit four of six targets in adverse winds, though because such an event is quite unlikely, it would still be reported as very bad confidence.

The researchers argue these competency reports can calibrate human trust in autonomy and improve mission performance. In a practical human-machine team, a supervisor could ask the aircraft whether it will complete the mission with a given battery reserve, and in calm winds it would report very good confidence while in adverse winds it would report very bad. Both platforms would report very good confidence of visiting the first target within 20 seconds, but both would report very bad confidence of visiting four or more targets, prompting a follow-up question about three targets, where calm winds yield very good confidence and adverse winds yield an even chance. The team also identified challenges for live platforms: a priori self-assessments can be invalidated by in-flight changes in weather or tasking, and fixed-wing aircraft cannot pause operation to re-assess the way ground robots can. The computational bottleneck lies in simulating mission rollouts, whose cost scales with time horizon, mission complexity, and the number of rollouts requested, motivating research into efficient sampling and rollout reuse. The work was partially supported by the Defense Advanced Research Projects Agency under contract HR001120C0032, and the authors suggest that validating competency self-assessments with live aerial platforms and humans in the loop is a key next step toward safer deployment of autonomous aircraft.

Subject of Research: Competency self-assessment enables a learning-based autonomous aircraft to quantify and communicate its mission capabilities to human supervisors.

Article Title: Competency self-assessment for a learning-based autonomous aircraft system

Article References: Conlon, N., Acharya, A., McGinley, J., Slack, T., Hirst, C. A., D’Alonzo, M., Hebert, M. R., Reale, C., Frew, E. W., Russell, R., & Ahmed, N. R. (2025). Competency self-assessment for a learning-based autonomous aircraft system. Frontiers in Aerospace Engineering, 4, Article 1454832. https://doi.org/10.3389/fpace.2025.1454832

Image Credits: AI Generated

DOI: 10.3389/fpace.2025.1454832

Keywords: Competency, self-assessment, learning-based, autonomous, aircraft, system, scientific research