NASA-built AI assistant taps verified aviation tools to cut hallucination risk

Air taxis, delivery drones, and other advanced air mobility vehicles are rapidly adding new layers of traffic to an already crowded sky, and the humans who manage that traffic are running out of attention to spare. Air traffic decision-makers now face a barrage of real-time weather updates, notices to air missions, traffic advisories, and flight data streams, all while fatigue, stress, and uneven expertise across shifts erode situational awareness. A team at NASA Ames Research Center believes large language models, deployed carefully, can help. In a new study published in Frontiers in Aerospace Engineering, researchers Aida Sharif Rohani, Stephen S. Clarke, and Aditya Das describe an LLM-based decision-support method that answers aviation professionals’ natural language questions by routing them through validated expert models and live data sources rather than relying on the model’s own statistical guesses.

The central problem the team set out to solve is hallucination, the well-documented tendency of language models to produce fluent but false answers. Research cited in the study, including a Nature paper showing that larger, more instructable language models become less reliable over time, underscores why this matters in a domain where a wrong answer can carry serious operational consequences. The researchers’ solution is architectural rather than statistical: instead of letting the model generate answers from internal knowledge, they make it act as a decision-making layer between the user and a library of nine aviation-validated tools, from trajectory prediction models to FAA document retrieval and an air traffic control simulator.

When a query arrives, the system first interprets the user’s intent, extracting key entities such as airport names, weather conditions, and traffic management initiatives. It then selects the appropriate tool or combination of tools, fills in required inputs from defaults or auxiliary plugins, executes the model, and synthesizes the output into a natural-language response that explicitly references the data sources, models, and assumptions used. If the user asks, for example, for the best runway configuration at JFK right now, the system calls an FAA API to fetch current and forecast weather and feeds that data into NASA’s rule-based runway configuration model, then explains the result with full transparency about what was used and what was assumed.

The nine-tool arsenal is a who’s who of operational aviation expertise. A machine learning model estimates aircraft performance parameters such as drag coefficients and weight from historical flight records to sharpen trajectory predictions. A rule-based model anticipates runway configuration changes driven by wind and closures. A DeBERTa-based natural language processing pipeline extracts traffic management decisions from FAA teleconference transcripts with near human-level F1 scores. Additional plugins tap the FAA’s ATCSCC operations plan advisories, an aviation route API, live METARs and TAFs from aviationweather.gov, NASA’s cloud-based live operational data services streaming thousands of flight updates per second, retrieval-augmented access to FAA manuals, and the open-source BlueSky air traffic control simulator.

To test whether an LLM could reliably navigate this constrained landscape, the team built a dataset of 1,000 manually drafted and verified aviation queries spanning five categories. Nearly 60 percent were straightforward single-tool queries, but the rest deliberately probed failure modes: 17.6 percent omitted required parameters, 10.7 percent presented edge cases or fictional scenarios, 7.3 percent phrased questions to resemble the wrong tool, and 4.6 percent required no tool at all. Aviation experts collaboratively established ground-truth labels, adopting the more conservative interpretation whenever they disagreed, and an independent professional drafted a separate 50-question held-out test set after prompt tuning was complete.

The results were striking. GPT-4o achieved an average correct selection rate of 95.2 percent and input selection accuracy of 96.1 percent across all nine tools, with several tools, including entity recognition, reaching a perfect 100 percent. The smaller GPT-4o-mini followed at 91.5 and 93.7 percent respectively. On the independent held-out set, the system scored 94 percent, suggesting the prompt engineering process did not overfit to the development data. The most persistent weakness was the weather tool, at 86.3 percent, where the models occasionally called the tool for general meteorological questions that should have been answered directly or declined.

Perhaps the most practically useful finding concerns when a smaller, cheaper model suffices. For tools with few operational constraints, such as weather, routes, live data, and FAA manuals, the performance gap between the two models was a negligible 0.8 to 1.5 percentage points. But for tools enforcing hard boundaries, like the runway configuration model that supports only three airports, the gap ballooned to 15.8 points, because GPT-4o was far better at declining out-of-scope queries while GPT-4o-mini tended to call the tool regardless. The authors suggest that lighter models may be viable for open-ended tool calling, while full-size models are preferable where scope enforcement is critical.

The team also ran a revealing baseline comparison against a plain GPT-4o without tool access. For real-time data sources, the baseline correctly declined everything, having no way to see current advisories or weather. Given identical raw flight data, it failed on structured queries like counting flights above a certain altitude or filtering by wake turbulence category, while the tool-grounded system answered correctly. Only on static regulatory questions did the baseline win on raw accuracy, at 95 percent versus 74 percent, but it produced one factual error by misidentifying the hijacking transponder code, precisely the kind of confident hallucination the grounded system avoids by returning no-information-found responses when content is absent from its corpus.

Multi-tool scenarios showcase the system’s fluidity. When a user asked about current weather and then the best runway configuration at an airport, the system retained the weather data in memory and fed it into the runway model. In another case, it fetched the latest FAA advisory affecting Boston, then used that context to recommend optimal routes from Orlando. When a user requested aircraft performance parameters for an Airbus A319 arriving at Dallas-Fort Worth, the system proactively retrieved weather first, then invoked the trajectory model, listing all its input assumptions. In a final demonstration, it translated an airline shorthand query into a live database count of airborne United flights, then instantiated one of those flights in the BlueSky simulator with live data and issued descent commands.

The work required substantial human effort: early tools needed more than 20 revision cycles of expert feedback before prompts stabilized, though later tools converged far faster. The authors are candid about limitations, noting that the 1,000-query dataset is modest, that the system cannot extrapolate beyond hard tool constraints, and that only GPT-based models were tested. Still, the study makes a compelling case that the future of AI in safety-critical domains may look less like an oracle and more like a well-trained librarian, one that knows exactly which expert to consult, what data to fetch, and when to admit it does not know. As advanced air mobility fills the skies, that kind of disciplined humility may prove to be the most valuable feature an AI assistant can offer.

Subject of Research: LLM-based decision support grounded in validated aviation models and real-time data for air traffic management

Article Title: A large language model–based method for improving decision-making in aviation

Article References: A large language model–based method for improving decision-making in aviation. (n.d.). https://doi.org/10.3389/fpace.2026.1808857

Image Credits: AI Generated

DOI: 10.3389/fpace.2026.1808857

Keywords: large language models, aviation decision support, air traffic management, NASA Ames, GPT-4o, hallucination, tool calling, advanced air mobility, prompt engineering, trajectory prediction, retrieval-augmented generation, BlueSky simulator