Foundation Models Become Driving Mentors in New Open-Source Autonomous Driving Platform

Autonomous driving researchers have long faced an awkward trade-off. The large vision-language models that can reason about a complicated traffic scene are far too slow to control a vehicle in real time, while the lightweight reinforcement learning agents that can react in milliseconds often lack the semantic understanding needed to navigate messy, human-filled streets. A new open-source platform called Found-RL, developed by researchers and described in a study published in Communications in Transportation Research, proposes a way to have both: it lets foundation models act as background mentors during training, transferring their knowledge into compact policies that remain fast enough for real-world deployment.

The central problem the team set out to solve is one of timescales. Reinforcement learning for end-to-end driving typically requires millions of interactions with a simulated environment, often running many parallel simulation workers at high frequency. Calling a multi-billion-parameter vision-language model at every simulation step would stall that loop entirely, turning a training run that should take hours into one that could take weeks. Yet without such a model, a reinforcement learning agent must discover everything about driving semantics, from yielding conventions to the meaning of traffic signals, through blind trial and error.

“Large vision-language models can provide useful driving knowledge, but directly calling them at every simulation step would make reinforcement learning training extremely slow,” says Sikai Chen, corresponding author of the study and assistant professor at the University of Wisconsin–Madison. “Found-RL is designed to bridge this gap by allowing foundation models to guide reinforcement learning in the background, without blocking the simulation process.” The key architectural insight is that a mentor does not need to answer instantly; it needs to answer eventually, and its advice needs to reach the right student at the right moment.

Found-RL achieves this with an asynchronous batch inference framework that decouples simulation rollout from model inference. Multiple simulation workers continuously generate driving observations and contextual information, which they push into a shared request queue rather than waiting on. A dedicated inference server collects these requests, processes them in batches to exploit the parallelism that modern accelerators handle best, and returns feedback asynchronously. When the feedback arrives, the platform matches it to the correct training sample and stores it in the replay buffer, where the reinforcement learning agent can learn from it. The agent never pauses to wait for the model; it simply keeps interacting with the driving environment while the foundation model works through the backlog of questions in the background.

“Found-RL treats the vision-language model as a scalable mentor rather than a real-time controller,” Chen explains. “The lightweight reinforcement learning policy still performs high-frequency control, while the foundation model provides semantic supervision during training.” This division of labor mirrors how human drivers learn: an instructor offers rich commentary and corrections during lessons, but the student behind the wheel makes every split-second decision alone. Once training ends, the mentor is no longer needed in the loop at all.

On top of this infrastructure, the researchers implemented two distinct forms of vision-language-model supervision. The first is action guidance, in which the model supplies expert-like driving actions that help the agent explore the vast space of possible behaviors more efficiently. Two mechanisms fold these suggestions into the learning process: Value-Margin Regularization and Advantage-Weighted Action Guidance, which weight the model’s advice so that it steers exploration without overriding what the agent has already learned from its own experience.

The second supervision channel is reward shaping based on CLIP, the contrastive vision-language model that scores how well an image matches a text description. Applying CLIP directly to continuous driving control is difficult, so the team introduced a conditional contrastive action-alignment reward. The method conditions text prompts on driving context, such as the vehicle’s current speed and the active navigation command, and then compares the agent’s chosen action against context-specific semantic action anchors. The result is a dense, interpretable reward signal that tells the agent not merely whether it reached a goal, but whether its behavior matched the semantic description of good driving in that particular situation.

The team evaluated Found-RL in CARLA, the widely used open-source driving simulator, across several standard autonomous driving benchmarks including Leaderboard and NoCrash. The results show consistent improvements over classical reinforcement learning algorithms in driving score, route completion, and safety-related metrics. In the Leaderboard benchmark, the DrQv2-CLIP agent achieved a Driving Score of 0.77 and a Success Rate of 57 percent, outperforming standard reinforcement learning baselines. The study also found that agents enhanced by vision-language models reduced unsafe behaviors such as vehicle collisions and red-light violations across several test settings, suggesting that the semantic feedback translates into measurably safer driving rather than merely higher scores.

Perhaps the most striking finding is the efficiency of knowledge transfer. Found-RL distills useful semantic knowledge from enormous foundation models into compact reinforcement learning policies: the resulting agent contains only 3.82 million parameters yet achieves driving performance competitive with models billions of times larger in parameter count. The lightweight policy reaches approximately 500 frames per second in forward-pass inference, compared with roughly 1 frame per second for several billion-parameter vision-language models. That three-orders-of-magnitude speed difference is exactly what separates a policy that could plausibly run on vehicle hardware from one confined to the data center. “These results suggest that foundation models do not necessarily need to drive the vehicle directly,” says Chen. “They can instead serve as teachers during training, helping smaller and faster policies learn safer and more efficient driving behaviors.”

The researchers have released Found-RL as an open-source framework, positioning it as infrastructure for future work at the intersection of foundation models, reinforcement learning, and autonomous driving. The authors outline several directions ahead: testing the platform on more challenging driving benchmarks, accelerating and quantizing vision-language-model inference to tighten the training loop further, combining multiple foundation models that each provide specialized feedback, and exploring lightweight vision-language models as policy backbones in their own right. “Autonomous driving requires both fast control and rich scene understanding,” Chen says. “Found-RL offers a practical way to combine these strengths by using foundation models to improve training while keeping the deployed driving policy efficient.” If the mentor-student paradigm holds up outside the simulator, the expensive reasoning of large models may end up living permanently in the training pipeline, while the vehicles on the road carry only what they learned from it.

Subject of Research: Integrating vision-language foundation models into reinforcement learning for efficient autonomous driving training

Article Title: Turning Foundation models into “driving mentors”: found-RL enables efficient reinforcement learning for autonomous driving

Article References: Turning Foundation models into “driving mentors”: found-RL enables efficient reinforcement learning for autonomous driving. (n.d.). Original publication

Image Credits: AI Generated

DOI: Not provided

Keywords: Found-RL, autonomous driving, reinforcement learning, vision-language models, foundation models, CARLA simulator, asynchronous inference, CLIP reward shaping, knowledge distillation, real-time policy inference, driving safety, open-source platform