The central problem the team set out to solve is one of timescales. Reinforcement learning for end-to-end driving typically requires millions of interactions with a simulated environment, often running many parallel simulation workers at high frequency. Calling a multi-billion-parameter vision-language model at every simulation step would stall that loop entirely, turning a training run that should take hours into one that could take weeks. Yet without such a model, a reinforcement learning agent must discover everything about driving semantics, from yielding conventions to the meaning of traffic signals, through blind trial and error.

“Large vision-language models can provide useful driving knowledge, but directly calling them at every simulation step would make reinforcement learning training extremely slow,” says Sikai Chen, corresponding author of the study and assistant professor at the University of Wisconsin–Madison. “Found-RL is designed to bridge this gap by allowing foundation models to guide reinforcement learning in the background, without blocking the simulation process.” The key architectural insight is that a mentor does not need to answer instantly; it needs to answer eventually, and its advice needs to reach the right student at the right moment.

Found-RL achieves this with an asynchronous batch inference framework that decouples simulation rollout from model inference. Multiple simulation workers continuously generate driving observations and contextual information, which they push into a shared request queue rather than waiting on. A dedicated inference server collects these requests, processes them in batches to exploit the parallelism that modern accelerators handle best, and returns feedback asynchronously. When the feedback arrives, the platform matches it to the correct training sample and stores it in the replay buffer, where the reinforcement learning agent can learn from it. The agent never pauses to wait for the model; it simply keeps interacting with the driving environment while the foundation model works through the backlog of questions in the background.

“Found-RL treats the vision-language model as a scalable mentor rather than a real-time controller,” Chen explains. “The lightweight reinforcement learning policy still performs high-frequency control, while the foundation model provides semantic supervision during training.” This division of labor mirrors how human drivers learn: an instructor offers rich commentary and corrections during lessons, but the student behind the wheel makes every split-second decision alone. Once training ends, the mentor is no longer needed in the loop at all.

On top of this infrastructure, the researchers implemented two distinct forms of vision-language-model supervision. The first is action guidance, in which the model supplies expert-like driving actions that help the agent explore the vast space of possible behaviors more efficiently. Two mechanisms fold these suggestions into the learning process: Value-Margin Regularization and Advantage-Weighted Action Guidance, which weight the model’s advice so that it steers exploration without overriding what the agent has already learned from its own experience.

The second supervision channel is reward shaping based on CLIP, the contrastive vision-language model that scores how well an image matches a text description. Applying CLIP directly to continuous driving control is difficult, so the team introduced a conditional contrastive action-alignment reward. The method conditions text prompts on driving context, such as the vehicle’s current speed and the active navigation command, and then compares the agent’s chosen action against context-specific semantic action anchors. The result is a dense, interpretable reward signal that tells the agent not merely whether it reached a goal, but whether its behavior matched the semantic description of good driving in that particular situation.

The team evaluated Found-RL in CARLA, the widely used open-source driving simulator, across several standard autonomous driving benchmarks including Leaderboard and NoCrash. The results show consistent improvements over classical reinforcement learning algorithms in driving score, route completion, and safety-related metrics. In the Leaderboard benchmark, the DrQv2-CLIP agent achieved a Driving Score of 0.77 and a Success Rate of 57 percent, outperforming standard reinforcement learning baselines. The study also found that agents enhanced by vision-language models reduced unsafe behaviors such as vehicle collisions and red-light violations across several test settings, suggesting that the semantic feedback translates into measurably safer driving rather than merely higher scores.

Perhaps the most striking finding is the efficiency of knowledge transfer. Found-RL distills useful semantic knowledge from enormous foundation models into compact reinforcement learning policies: the resulting agent contains only 3.82 million parameters yet achieves driving performance competitive with models billions of times larger in parameter count. The lightweight policy reaches approximately 500 frames per second in forward-pass inference, compared with roughly 1 frame per second for several billion-parameter vision-language models. That three-orders-of-magnitude speed difference is exactly what separates a policy that could plausibly run on vehicle hardware from one confined to the data center. “These results suggest that foundation models do not necessarily need to drive the vehicle directly,” says Chen. “They can instead serve as teachers during training, helping smaller and faster policies learn safer and more efficient driving behaviors.”

The researchers have released Found-RL as an open-source framework, positioning it as infrastructure for future work at the intersection of foundation models, reinforcement learning, and autonomous driving. The authors outline several directions ahead: testing the platform on more challenging driving benchmarks, accelerating and quantizing vision-language-model inference to tighten the training loop further, combining multiple foundation models that each provide specialized feedback, and exploring lightweight vision-language models as policy backbones in their own right. “Autonomous driving requires both fast control and rich scene understanding,” Chen says. “Found-RL offers a practical way to combine these strengths by using foundation models to improve training while keeping the deployed driving policy efficient.” If the mentor-student paradigm holds up outside the simulator, the expensive reasoning of large models may end up living permanently in the training pipeline, while the vehicles on the road carry only what they learned from it.

Subject of Research: Integrating vision-language foundation models into reinforcement learning for efficient autonomous driving training

Article Title: Turning Foundation models into “driving mentors”: found-RL enables efficient reinforcement learning for autonomous driving

Article References: Turning Foundation models into “driving mentors”: found-RL enables efficient reinforcement learning for autonomous driving. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: Found-RL, autonomous driving, reinforcement learning, vision-language models, foundation models, CARLA simulator, asynchronous inference, CLIP reward shaping, knowledge distillation, real-time policy inference, driving safety, open-source platform

The core insight of the work is that the models that power modern chatbots, known as vision-language models, can do more than describe a scene. They can be taught to make bounded, auditable safety decisions. But the researchers found that off-the-shelf versions of these models fail at the task in a dangerous and instructive way. When general-purpose models were asked, without any task-specific training, to watch short video clips of an intersection and recommend an action, the larger seven-billion-parameter model intervened on nearly every clip it saw, including clips of completely normal traffic where the correct answer was to do nothing. Its apparent accuracy on hazardous events was an artifact of always sounding the alarm. The smaller three-billion-parameter model, meanwhile, produced outputs in the wrong format, so the signal controller could not execute them at all. For a safety layer, the decisive failure is not missing a hazard but crying wolf on every quiet street corner, eroding both traffic flow and trust.

To fix this, the team built a training dataset of 1,020 video clips generated inside the CARLA driving simulator, each roughly two to three seconds long and captured by two fixed virtual cameras whose views are stacked into a single composite image of the intersection. The clips cover nine combinations of weather and lighting, from clear noon to hard rain at night, and fall into three classes: jaywalking, red-light running, and normal traffic. Every clip was labeled automatically by a deterministic expert policy that maps each event and signal state to an ideal intervention. A jaywalker triggers a long all-red hold of eight to ten seconds, calibrated to the geometry of the crosswalks, which span about 14.5 to 15 meters; a red-light runner triggers a shorter hold of five seconds for cars and six for heavy vehicles, enough to clear the intersection before conflicting movements are released; normal traffic triggers no action at all.

One of the most technically interesting parts of the pipeline addresses a subtle data problem. The object detector that anchors the system, a YOLOv10-l model, performs well in daylight but nearly collapses at night and in heavy rain. In the first collection pass, the detector found around 40 jaywalking pedestrians per daytime condition but only six on a clear night and a single one under hard rain at night. Because the pedestrians were scripted actors whose positions the simulator knows exactly, the researchers projected ground-truth positions onto the camera images using pinhole camera geometry, recovering correct labels for 128 of the 348 jaywalking samples. Crucially, the projection changes only the labels, not the video itself, so the model still has to learn to recognize faint, barely visible hazards from raw pixels alone.

The fine-tuning itself is remarkably efficient. The researchers started with Qwen2.5-VL-3B-Instruct, a compact open model, and applied low-rank adaptation, a technique that freezes the original network and injects a small number of trainable update matrices into its attention and feed-forward layers. Only 37.15 million parameters, less than one percent of the model, were trained, and the vision encoder was left untouched. Training took about 40 minutes on a single workstation GPU, with loss decaying smoothly from 1.13 to 0.004 over three passes through the data. The model was trained to emit a strict JSON decision document: whether to intervene and why, how long to hold all-red, which legal signal combination to run, per-phase green times, safety notes, and a confidence level. Every field is range-checked by a separate rule-constrained controller, which rejects anything malformed, out of bounds, or inconsistent with traffic-engineering constraints. The AI never touches the signal directly.

The evaluation was deliberately harsh. Three of the nine weather conditions, wet sunset, clear night, and hard rain at night, were withheld entirely from training, and the fine-tuned model was tested on 313 clips from those unseen conditions. It achieved perfect schema validity, perfect adherence to controller bounds, and a perfect intervene-or-do-nothing decision on every single clip, with a mean hold-time error of just 0.15 seconds. Its false-positive rate on the 108 normal-traffic clips was zero, with a statistical upper bound of 3.4 per hundred. The zero-shot seven-billion-parameter baseline, by contrast, scored only 0.655 on decision accuracy, exactly the fraction of event clips in the test set, because it intervened on all of them, and its hold-time errors averaged over four seconds. Notably, a seven-billion-parameter model fine-tuned with the identical recipe performed no better than the three-billion one, and the smaller model was also faster, with a median decision time of 2.77 seconds versus 3.20 seconds for its larger zero-shot rival. Adaptation, not scale, is what makes these models usable inside a control loop.

The system also generalized beyond its training environment. At a second simulated intersection in a different virtual town, with different road geometry, surroundings, and camera positions, and with no retraining whatsoever, the fine-tuned model again made the correct decision on all 135 test clips. The only degradation was in hold timing: it recommended nine-second holds where the site-specific expert target was eight seconds, a one-second error that still fell within the mandated eight-to-ten-second band. Textual explanations produced at the new site referred to its actual crosswalks and approaches rather than echoing the training scene, suggesting the model carried across an imitated policy rather than a memory of one location’s appearance.

In closed-loop operation, where the full sensing, reasoning, and control stack ran live against scripted violation scenarios, the safety benefits were substantial but came with a measurable cost. Compared with blind fixed-time signal control, the ground-truth pedestrian exposure fraction, the share of total crosswalk-occupancy time overlapping a conflicting green, was 67.6 percent lower at low demand and 73.0 percent lower at medium demand, with every random seed favoring the AI layer. Against a stronger actuated pedestrian-clearance baseline, the reductions were 55.4 and 71.6 percent. An analysis showed the gains came not merely from serving less conflicting green but from shifting that green away from moments when pedestrians were actually present. The price, at medium demand, was a 105.9 percent increase in average stopped delay and a 13.8 percent throughput drop, though the researchers caution that the scripted scenarios inject hazardous events far more often than real intersections experience, inflating the intervention frequency and its delay burden by construction.

The study is equally candid about its limits, and the most consequential one lies upstream of the AI. At night, the object detector recovers only about 10.7 percent of ground-truth crosswalk occupancy from the same imagery, compared with 62.6 percent in daylight, and under clear-night conditions the closed-loop safety improvement shrank to a statistically insignificant 15.2 percent. A semantic layer cannot protect a pedestrian it cannot see. All results come from simulation, with safety-critical events injected by script rather than observed in natural traffic, so no claim of real-world transfer is made. Still, the authors argue the central conclusion stands: domain adaptation, not model scale, is what allows a vision-language model to operate reliably within a controller’s constraints, and the path to deployment now runs through weather-robust perception, sensor fusion, and shadow-mode validation on real traffic cameras. If that perception gap closes, the vision of intersections that watch, understand, and briefly freeze to protect the most vulnerable road users moves a significant step closer to reality.

Subject of Research: A lightweight vision-language AI safety layer for pedestrian protection at signalized intersections

Article Title: A Fine-Tuned Lightweight Vision-Language Safety Layer with Weather-Generalising Decisions at Signalized Intersections

Article References: Afshari, A., Lee, J., & Mashal, Y. (2026). A Fine-Tuned Lightweight Vision-Language Safety Layer with Weather-Generalising Decisions at Signalized Intersections. Machine Learning with Applications, 26, Article 101032. https://doi.org/10.1016/j.mlwa.2026.101032

Image Credits: AI Generated

DOI: 10.1016/j.mlwa.2026.101032

Keywords: vision-language models, traffic signal control, pedestrian safety, fine-tuning, LoRA, CARLA simulator, computer vision, intelligent transportation systems, weather generalization, all-red hold, machine learning, road safety

That combination, however, creates a blind spot. When data flows from a ROS 2 node into an AUTOSAR AP application and back again, each platform keeps its own records of what happened. ROS 2 offers tracing hooks that capture communication events, while AUTOSAR AP applications typically rely on ara::log, the platform’s standardized logging facility, which was never designed to serve as a precision timing instrument. Stitching these two heterogeneous record streams into a single, coherent picture of end-to-end latency has been a stubborn engineering problem. Developers prototyping a new perception or planning module can easily discover that something feels slow without being able to say whether the delay arose in sensor reception, in cross-platform message translation, in the detection algorithm itself, or in the final command output.

A research team led by Professor Takuya Azumi of the Graduate School of Science and Engineering at Saitama University, working with Astemo, Ltd., has now pushed a solution to that problem much closer to practical reality. The team had previously developed CART, the Combined AUTOSAR AP and ROS 2 Tracing Framework, which integrates trace information from both platforms into a unified timeline. But the original evaluation of CART was confined to a small-scale application configuration, leaving open a question that matters enormously to anyone building real vehicles: would the framework still deliver useful end-to-end latency analysis when confronted with the component count, data volume and complexity of a genuine autonomous-driving stack? The answer, published on August 6, 2026 in the IEEE Open Journal of the Industrial Electronics Society under the title “Evaluation Platform for Tracing of Autonomous Driving System Combined AUTOSAR AP and ROS 2” (DOI: 10.1109/OJIES.2026.3721135), is a carefully qualified yes.

To reach that answer, the researchers built a cloud-based evaluation platform that assembles four major pieces of software into one instrumented pipeline. The high-fidelity CARLA simulator generates realistic LiDAR point-cloud data for a simulated vehicle. Those point clouds flow into Autoware, the well-known open-source autonomous-driving software stack built on ROS 2, which handles the higher-level driving logic. From Autoware, sensor data pass across the platform boundary into an AUTOSAR AP application responsible for object detection, with detection results routed back to Autoware through protocol bridges so that vehicle-control commands can be generated. CART sits alongside this entire chain, ingesting ROS 2 traces and AUTOSAR AP logs and reconstructing the processing path from the moment simulated sensor data are received to the moment a control command emerges at the other end.

The results of the evaluation speak to both the completeness and the realism of the setup. CART reconstructed the entire expected end-to-end communication topology with 100 percent coverage, meaning that no segment of the processing path across the ROS 2–AUTOSAR AP boundary was lost to the tracing process. Under the evaluated workload, the platform sustained a point-cloud reception callback rate of approximately 33 Hz, a figure that reflects the kind of continuous, high-frequency sensor processing real autonomous-driving software must handle. Crucially, all of this was achieved with low tracing overhead, so the act of measurement did not materially distort the behavior being measured — a perennial hazard in performance analysis.

The framework also demonstrated an ability that fixed communication diagrams cannot provide: following dynamic control behavior. In the evaluated scenarios, CART distinguished between processing paths in which object detection triggered a stop command and paths in which no actuation command was issued at all. That distinction matters because autonomous-driving logic is not a static pipeline; it branches depending on what the sensors perceive. A tracing tool that can only follow fixed routes would miss precisely the situations developers care most about — the moments when the vehicle’s software decides to change what it is doing. By capturing those transitions, CART showed it can trace changes in control logic as well as ordinary data flow.

Overhead, the quiet killer of any tracing scheme, was quantified in two separate tests. In a small-scale test, adding ara::log-based instrumentation introduced little CPU load and roughly 0.03 MiB of memory usage, numbers small enough to be negligible for most development purposes. In a separate offline benchmark, converting logs representing one million send/receive pairs into trace data took approximately one second, suggesting that the framework can scale to the volume of logging that a full autonomous-driving stack generates without becoming an analysis bottleneck in itself. The authors assessed the platform at Technology Readiness Level 6, based on its demonstration in a relevant simulation environment, while noting that validation on real hardware remains future work.

“This study is important because autonomous-driving software is becoming increasingly complex, with different software platforms working together within the same system,” said Shunsuke Ito, a master’s student at Saitama University’s Graduate School of Science and Engineering and the corresponding author of the study. “By integrating trace information from AUTOSAR AP and ROS 2, our platform makes it possible to follow the processing path from sensor input to the generation of a control command as a single end-to-end sequence. This allows developers to see more clearly where latency occurs across platform boundaries.” The point is not merely academic. Latency budgets in autonomous driving are unforgiving: a perception-to-control chain that overshoots its deadline can degrade the freshness of the vehicle’s world model, and the difference between a safe response and a dangerous one can hinge on tens of milliseconds.

Ito emphasized that the leap from small test applications to realistic workloads was the heart of the contribution. “It is not enough to confirm that a tracing method works with a small test application,” he explained. “Autonomous-driving systems process large amounts of sensor data through many interconnected software components. By combining CARLA, Autoware, AUTOSAR AP, and ROS 2 in a cloud environment, we were able to test CART under more realistic conditions and show that detailed latency analysis remains possible without imposing a large additional processing burden.” For engineering teams practicing rapid prototyping, that means performance bottlenecks can now be identified during the design-and-test cycle, before software is transferred to physical vehicle hardware — when fixes are cheapest and iteration is fastest.

The implications extend to the broader industry shift toward Software-Defined Vehicles, in which functionality is increasingly delivered and updated through software rather than hardware. By providing a unified view of processing across heterogeneous platforms, unified tracing could shorten development and validation cycles and make complex vehicle software systematically evaluable. The team is candid about the road ahead: practical deployment will require validation with real electronic control units or hardware-in-the-loop systems, clock synchronization across multiple computing systems, larger-scale testing, and further automation of cross-platform message correlation. Looking forward, the researchers plan to extend the platform to lighter-weight simulation environments, incorporate OpenSCENARIO-based systematic testing, automate message linking using standard SOME/IP identifiers, and ultimately evaluate CART on real hardware. “If developers can trace the behavior of those systems across different platforms and identify performance problems earlier, they may be able to shorten development and validation cycles while making complex vehicle software easier to evaluate systematically,” Ito said. “In the future, we hope to extend CART from cloud-based simulation to real ECUs and further automate the tracing process. If these capabilities can be incorporated into practical vehicle-development workflows, unified tracing could become a useful tool for developing autonomous-driving and software-defined vehicle functions more efficiently and reliably.”

Subject of Research: End-to-end latency tracing across mixed ROS 2 and AUTOSAR AP autonomous-driving software platforms

Article Title: New tracing platform pinpoints delays across complex autonomous-driving software

Article References: New tracing platform pinpoints delays across complex autonomous-driving software. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: autonomous driving, CART, AUTOSAR AP, ROS 2, Autoware, CARLA simulator, latency tracing, software-defined vehicles, Saitama University, ara::log, cloud evaluation platform, IEEE Open Journal of the Industrial Electronics Society