New Framework Uses Language Models to Evaluate Autonomous Vehicle Traffic Intelligence

Researchers at Tongji University have developed a new evaluation framework that assesses how intelligently high-level autonomous vehicles behave during complex traffic interactions. The method, termed the Subjective-Objective Mapping Evaluation framework, couples large language model reasoning with measurable driving metrics to evaluate comprehensive vehicle behavior rather than isolated driving functions.

The development comes as automated driving systems transition from experimental pilot projects toward broader commercial deployment on public roadways. Although autonomous vehicles may perform successfully on standard technical examinations, vehicles that pass conventional benchmarks can still exhibit driving patterns that feel abrupt, hesitant, or disruptive to surrounding road users in real-world traffic conditions.

According to study corresponding author and Tongji University research professor Peng Hang, the assessment of high-level autonomous driving systems must extend beyond fundamental collision avoidance. Hang explained that an automated vehicle’s behavior should also be evaluated on whether it maneuvers smoothly, fulfills driving objectives efficiently, interacts appropriately with other road users, and prevents unnecessary disruptions to the surrounding traffic stream.

To capture the multidimensional nature of driving quality, the evaluation framework examines five complementary behavioral aspects: safety, comfort, efficiency, social interaction, and impact on surrounding traffic. By analyzing these five areas simultaneously, the system seeks to replicate the broad and nuanced evaluations typically provided by human transportation specialists.

The research team trained and evaluated the framework using a collection of 5,200 valid interaction segments constructed from the Waymo Open Dataset. These scenarios encompassed common driving challenges, including vehicle-following maneuvers, lane merging, and intersection crossing, while researchers also augmented the dataset with physically feasible low-score interaction samples to ensure adequate representation of rare but critical poor-performance situations.

Because large language models cannot directly interpret raw sensor trajectories and digital map inputs, the investigators engineered a dedicated Data-Language Mapping Module. This system converts vehicle motion states, abnormal maneuvers, and key traffic events into structured natural-language descriptions, which are then combined with traffic rules, driving guidance, and domain knowledge to allow the language model to analyze driving interactions in an expert-like manner.

The subjective assessments generated by the language model subsequently serve as supervisory training labels for an attention-based neural network. Instead of relying on a rigid set of manually assigned scoring weights, the neural network learns nonlinear interdependencies among ten quantitative performance metrics, recognizing that travel efficiency, motion smoothness, vehicle safety, and social behaviors frequently influence one another in interconnected ways.

To assess whether the evaluations generated by the language model mirrored human analytical capabilities, the researchers conducted a Turing Test with ten doctoral students specializing in autonomous driving. Evaluated across 66 interaction samples, the language model’s assessments achieved an average passing rate of 86.75 percent, indicating that the expert reviewers frequently judged the automated evaluations to be human-generated or were unable to reliably distinguish their origin.

In comparative experimental testing, the proposed language model evaluation strategy exhibited the closest overall agreement with human expert ratings among tested configurations, recording a mean absolute error of 0.763, a Pearson correlation coefficient of 0.863, and the lowest score-distribution divergence. The downstream attention-based neural network demonstrated strong predictive precision for the model-derived intelligence scores, achieving a mean absolute error of 0.068 and an R-squared value of 0.880.

The researchers further validated the approach by testing five autonomous-driving algorithms across six scenario-based evaluations from the OnSite Autonomous Driving Challenge. The resulting rankings aligned closely with human expert appraisals and demonstrated clear advantages over traditional analytic hierarchy process methods, notably by identifying instances where high safety ratings obscured underlying deficiencies in passenger comfort, travel efficiency, or interactive behavior.

Hang highlighted that the framework is designed to provide a scalable and consistent evaluation mechanism for scenario-based testing and vehicle development rather than replacing human specialists in every decision. The authors noted that the current implementation focuses primarily on trajectory-level data and high-level behavioral indicators, with future investigations planned to incorporate multimodal inputs such as perception data and video streams across diverse regions, traffic cultures, and testing environments.

In evaluating autonomous driving systems, the Tongji University researchers identified a key operational limitation in conventional fixed-weight scoring methods such as the analytic hierarchy process. Standard assessment frameworks typically assign predetermined numerical weights to aggregate separate driving metrics. Under these rigid schemes, a vehicle can achieve a high overall score simply by satisfying basic collision-avoidance criteria, even if it causes substantial traffic delay or makes uncomfortable, jerky maneuvers. By treating safety, motion smoothness, travel efficiency, and social coordination as independent variables, fixed-weight models fail to capture the complex behavioral trade-offs inherent in real traffic. As a result, traditional benchmarks often struggle to separate genuinely competent automated driving systems from vehicles that avoid collisions through disruptive or overly hesitant driving patterns.

The Subjective-Objective Mapping Evaluation framework addresses these trade-offs by employing an attention-based neural network to model nonlinear relationships across ten quantitative metrics. Rather than relying on arbitrary manual weights, the network learns how objective driving indicators relate to the holistic judgments made by human experts observing vehicle interactions. Empirical analysis revealed that safety and comfort represent the two most influential dimensions for distinguishing different levels of autonomous driving intelligence. Furthermore, the findings confirmed that each of the five evaluated dimensions—safety, comfort, efficiency, social interaction, and impact on surrounding traffic—supplies distinct, essential information, demonstrating that thorough behavioral assessment requires evaluating both the vehicle’s immediate motion and its broader footprint on surrounding traffic flow.

To validate this approach, the researchers applied the framework across six scenario-based evaluations in the OnSite Autonomous Driving Challenge, testing five separate autonomous driving algorithms. The resulting rankings aligned closely with human expert appraisals, while offering clear diagnostic advantages over the analytic hierarchy process. Specifically, the model successfully identified cases where strong safety scores masked notable deficiencies in passenger comfort, travel efficiency, or interactive quality. This ability to expose hidden performance flaws highlights how learned nonlinear mapping directly captures real-world driving trade-offs, preventing autonomous systems from receiving inflated ratings when their collision avoidance comes at the expense of passenger comfort or surrounding traffic flow.

A vital technical component enabling the framework’s reasoning is the Data-Language Mapping Module, designed to overcome the inability of large language models to interpret raw trajectories and map data. The module converts key interaction events, vehicle motion states, and abnormal movements into structured natural-language descriptions. These textual summaries are then combined with autonomous driving knowledge, traffic rules, and driving-behavior guidance. By transforming continuous physical trajectories into structured natural language, the module provides the language model with the necessary contextual foundation to analyze multi-vehicle traffic encounters and produce qualitative, expert-like assessments of driving intelligence.

The empirical training of the framework used 5,200 valid interaction segments constructed from the Waymo Open Dataset, encompassing routine urban driving maneuvers such as vehicle following, lane merging, and intersection crossing. Because naturalistic driving logs consist predominantly of uneventful driving trajectories that lack sufficient representations of rare but critical failures, the researchers augmented the dataset with physically feasible low-score interaction samples. This targeted data expansion ensured balanced coverage of poor-performance driving cases, allowing both the large language model and the downstream neural network to establish robust evaluation boundaries across both competent and problematic driving behaviors.

To determine whether the large language model’s synthetic evaluations resembled human reasoning, the researchers conducted a Turing Test with ten doctoral students specializing in autonomous driving. Across 66 interaction samples, the automated evaluations achieved an average passing rate of 86.75 percent, indicating that the expert reviewers frequently judged the model-generated evaluations to be human-generated or could not reliably distinguish their source. In additional comparative trials, the proposed language model strategy achieved the closest overall agreement with human expert ratings, obtaining a mean absolute error of 0.763, a Pearson correlation of 0.863, and the lowest score-distribution divergence from expert judgments. Furthermore, the downstream attention-based neural network achieved a mean absolute error of 0.068 and an R-squared value of 0.880 when predicting the language model’s derived intelligence scores.

Regarding practical application, corresponding author Peng Hang emphasized that the framework is intended to provide a scalable, consistent mechanism for translating high-dimensional driving behavior into comprehensive evaluations that support scenario-based testing and vehicle engineering, rather than seeking to replace human experts in every decision. The authors noted that the current implementation focuses primarily on trajectory-level interactions and high-level behavioral indicators. To broaden its diagnostic scope, planned future investigations will incorporate richer multimodal inputs, such as raw video feeds, perception system outputs, and detailed environmental contexts, while expanding testing across additional driving datasets, geographical regions, traffic cultures, and evaluation platforms.

The study was published in Communications in Transportation Research under digital object identifier 10.26599/COMMTR.2026.9640041, with formal news release dissemination occurring on November 6, 2026, following the article’s publication date on September 30, 2026. Established in 2021 with academic support from Tsinghua University and the China Intelligent Transportation Systems Association, the peer-reviewed journal is led by Editors-in-Chief Professor Xiaobo Qu of Tsinghua University and Professor Xiaopeng Li of the University of Wisconsin–Madison. Published by Tsinghua University Press on the SciOpen platform from Volume 6 in 2026, the journal serves as an international platform for showcasing original research and review articles on emerging transportation systems, fostering academic exchange between China and the global community.

Subject of Research: Large language model framework for evaluating autonomous vehicle traffic intelligence

Article Title: Can LLMs help judge how intelligently autonomous vehicles behave in traffic?

Article References: Can LLMs help judge how intelligently autonomous vehicles behave in traffic?. Original publication (Bibliographic details unavailable.)

Image Credits: AI Generated

DOI: Not provided

Keywords: autonomous vehicles, artificial intelligence, large language models, traffic safety, machine learning, Tongji University, transportation technology, neural networks, LLMs, help, judge, intelligently