Slow-drift attacks can poison autonomous vehicle perception, and a language model watchdog fights back, study finds

A new study reports that the temporal memory inside bird’s-eye-view (BEV) perception systems, the persistent latent state that modern autonomous vehicle models update frame by frame, can be quietly corrupted by a sequence of tiny perturbations that each stay below the thresholds of existing defenses. The work, published in Discover Vehicles, formalizes this threat as Slow-Drift Temporal Poisoning (SDTP) and shows that on the BEVFormer-T architecture, frame-wise perturbations bounded by a small digital L∞ budget of ε ≤ 2/255 reach an 84.3% Temporal Drift Success Rate by frame 32. The paper also introduces SemantiGuard, a cross-modal defense that cross-checks BEV detections against a frozen vision-language model’s scene description and rolls back the latent memory when the two diverge, cutting the Temporal Drift Success Rate from 91.2% to 13.8% under the primary evaluation configuration.

The authors describe world-model BEV perception as a dominant paradigm for unified 3D scene understanding in autonomous vehicles. Architectures such as BEVFormer, BEVDet4D, and BEVFusion maintain a persistent latent BEV state, a compact spatiotemporal representation of the environment that is updated recurrently at each timestep via temporal attention mechanisms. This temporal aggregation improves detection of occluded objects, depth reasoning, and trajectory estimation. But the same recurrent design creates a dependency that can be systematically exploited: the detection output at time t is not solely dependent on instantaneous observations but is informed by accumulated temporal context spanning frames t-K to t.

SDTP exploits that dependency through a Backward Temporal Gradient (BTG) optimization procedure. Standard projected gradient descent optimizes a single-frame perturbation; SDTP instead optimizes a sequence of T perturbations jointly by backpropagating through the full temporal recurrence of the BEV model, using backpropagation through time to compute gradients with respect to each frame-wise perturbation. The adversarial objective drives the BEV memory toward a target state corresponding to object disappearance while a temporal smoothness penalty constrains per-frame variations in the BEV state so that inter-frame deviations remain below the detection threshold of temporal anomaly defenses. The target memory state is instantiated per attack instance by masking the BEV feature-grid cells and query embeddings within the footprint of the target bounding box in the model’s own clean memory state, operationalizing target disappearance as a concrete, model-consistent target.

The empirical pattern is stark. At a fixed perturbation budget of ε = 2/255, the Temporal Drift Success Rate for BEVFormer-T rises from 52.4% at a temporal window of T = 16 to 84.3% at T = 32, which the authors present as strong empirical evidence for the slow-drift accumulation effect. BEVFormer-T consistently shows the highest vulnerability among the four evaluated architectures, which the authors attribute to its strong temporal self-attention mechanism maintaining a persistent and highly influential memory state. BEVFusion, which fuses LiDAR with camera inputs, exhibits comparatively lower TDSR values, reaching 70.2% at ε = 2/255 and T = 32, suggesting that multi-sensor fusion alone does not constitute a sufficient defense against temporally accumulated adversarial perturbations.

Because increasing the temporal horizon simultaneously increases both the number of perturbed frames and the number of optimization variables, the authors ran two controls to isolate genuine temporal-memory poisoning from a generic multi-frame perturbation effect. In one, an independent single-frame PGD attack is applied to each of the T frames separately, without joint optimization through the temporal recurrence. In the other, SDTP’s jointly optimized perturbations are applied exactly as attack-optimized, but the BEV memory is reset to its clean value at every frame, removing temporal accumulation while keeping the per-frame perturbation content identical. The gap between SDTP and the strongest control at matched settings is 25.6 percentage points, which the authors say supports the interpretation that accumulated temporal-memory drift, rather than merely the number of perturbed frames, drives the attack’s effectiveness.

Why do existing temporal defenses miss this? Methods such as PhySense, PRBI, and ADAV predominantly assess consistency across consecutive frames using BEV representations or LiDAR point clouds, calibrated with single-step or pairwise inter-frame anomaly thresholds derived from benign driving conditions. Under nominal conditions, clean sequences typically exhibit Jaccard similarity scores above 0.85, whereas conventional single-frame adversarial attacks such as BIM, C&W, and PGD induce pronounced reductions to approximately 0.2, triggering detection. SDTP instead integrates a temporal smoothness regularization term within the BTG optimization, explicitly constraining per-frame variations in the BEV state so that inter-frame deviations remain below the detection threshold, approximately 0.12 under PRBI-calibrated settings, while still facilitating gradual accumulation over time. The authors frame this as a difference in threat-model coverage rather than a general deficiency of feature-space or sensor-space consistency checking.

SemantiGuard attacks the problem from a different direction. At each verification cycle, executed by default once every 4 frames at 25 Hz, a frozen vision-language model operating under 4-bit integer quantization generates a natural-language scene description from the same camera stream consumed by the attacked BEV model, with no assumption of a separately secured input channel. A lightweight JSON extraction head, fine-tuned on nuScenes-QA annotations, parses the response into a structured object inventory. A Semantic Divergence Scorer then cross-checks this description against the BEV detection output, computing a weighted combination of object-level set discrepancy and spatial misalignment, with weights w1 = 0.7 and w2 = 0.3 calibrated on a held-out subset of the official nuScenes validation split.

The defense operates through a three-tier response mechanism driven by calibrated thresholds. When the divergence score is below 0.20, the system proceeds under nominal operation. For intermediate values between 0.20 and 0.40, an Alert tier forwards detection outputs to downstream planning modules with attenuated confidence scores, encouraging more conservative decision-making. When divergence exceeds 0.40, the Rollback tier reverts the BEV latent memory to a prior state 8 frames back, a depth selected through a sensitivity sweep in which Defense Recall under Drift peaked at 82.3% at Kr = 8, while deeper rollbacks of 16 or 32 frames yielded no further improvement and progressively worse clean performance degradation. The rollback discards corrupted temporal information, allowing the perception system to reinitialize and progressively reconstruct a consistent BEV representation from subsequent clean observations.

Under the primary SDTP-L configuration (ε = 4/255, T = 32) on BEVFormer-T, SemantiGuard achieves the lowest TDSR of 13.8%, down from 91.2% without defense, a reduction of 77.4 percentage points, while attaining the highest Defense Recall under Drift of 81.7%. Adversarial training using standard PGD, the next strongest baseline, reduces TDSR only to 49.6% with a DRD of 46.0%, and incurs the highest clean performance degradation at −3.7 mAP. SemantiGuard maintains a more favorable balance with a CPD of −2.1 mAP, residual degradation the authors attribute to false-positive interventions such as rollback events triggered by legitimate rapid scene transitions like highway merging. The defense also generalizes across architectures on nuScenes, reducing TDSR to 13.8% for BEVFormer-T, 16.2% for BEVDet4D, 11.4% for M-BEV, and 10.7% for BEVFusion, an architecture-agnostic property the authors attribute to operating in the semantic domain of scene understanding rather than in BEV feature space.

The authors are explicit about the limits of these findings. All evaluation is open-loop, on fixed, pre-recorded nuScenes validation sequences, with no Waymo Open results reported, so the metrics should be read as perception-layer robustness measures, not estimates of downstream safety outcomes such as collision rate. The perturbation budget ε ≤ 2/255 is described as a standard proxy for low magnitude, not a validated measure of human-perceptual imperceptibility or physical-world realizability, and physically constrained configurations were evaluated in simulation only. Under an adaptive adversary that jointly optimizes the attack against both the BEV model and the semantic verifier, TDSR rises to 37.4% under configuration-matched comparisons, indicating residual robustness but also the need for further hardening. The 7B-parameter V2LM occupies roughly 3.5–4 GB even at 4-bit quantization, its 14 ms latency was measured in isolation, and concurrent-execution contention with the perception pipeline remains an open measurement gap. The authors identify closed-loop validation as the top priority for future work and treat physical-world feasibility as an open question rather than a demonstrated capability.

Subject of Research: Adversarial attacks on temporal memory in BEV perception for autonomous vehicles and a vision-language-model-based cross-modal defense

Article Title: Slow Drift Temporal Poisoning attacks and vision language model guided defense for BEV perception in autonomous vehicles

Article References: Hallur, S., Baskaran, G., Kattoju, H. K., Gajendra, B. V., & Beechu, S. G. (2026). Slow Drift Temporal Poisoning attacks and vision language model guided defense for BEV perception in autonomous vehicles. Discover Vehicles, 2(1), Article 44. https://doi.org/10.1007/s44465-026-00048-7

Image Credits: AI Generated

DOI: 10.1007/s44465-026-00048-7

Keywords: autonomous vehicles, BEV perception, adversarial attacks, temporal poisoning, vision-language models, SemantiGuard, BEVFormer, nuScenes, machine learning security, latent memory rollback, TemporalAdvBEV, open-loop evaluation