How AI Learned to See Roads the Way Humans Do

Every day, engineers, city planners, and self-driving car developers stare at the same question: is this road safe? It sounds simple, but the answer depends on who you ask and where the road is. A stretch of asphalt with no sidewalk might look like a death trap in downtown Manhattan, yet be perfectly normal on a country lane. Now, a team of researchers has built an artificial intelligence framework that finally understands this difference — and it beats some of the world’s most powerful AI models at its own game.

The study, published in Frontiers in Future Transportation, introduces CARSA, the Context-Aware Road Safety Assessment framework. Unlike conventional computer vision systems that simply detect objects — cars, pedestrians, traffic signs — CARSA interprets what those objects actually mean for human safety in a given environment. The result is a system that matched the judgment of a panel of human experts with 92.38 percent accuracy, while giant vision-language models like CLIP and Qwen2.5-VL managed barely 20 to 30 percent under the same test conditions.

The problem the researchers tackled is one of the most stubborn in intelligent transportation systems. Perceived road safety — how safe a road feels to the people using it — is fundamentally different from measured safety, which relies on crash statistics and infrastructure audits. Perception is subjective, contextual, and shaped by how humans interpret visual cues. A computer can identify a sidewalk with pixel-perfect precision, but deciding whether that sidewalk makes a street feel safe requires understanding the entire scene in its environmental context.

To build a fair test, the team started with the Mapillary Vistas dataset, a massive collection of street-level images with dense pixel-level annotations covering roads, sidewalks, vehicles, pedestrians, and infrastructure from around the world. Rather than randomly sampling images, they developed a semantic-richness-guided selection pipeline. The original 66 semantic categories were consolidated into 15 road-safety-oriented classes — including sidewalks, lane markings, curbs, barriers, traffic signs, traffic lights, poles, pedestrians, cyclists, vehicles, vegetation, buildings, and crosswalks. Each image received a richness score based on how many safety-relevant categories appeared in the scene, weighted by their importance to human safety perception.

From 20,000 images, the pipeline stratified scenes into high, medium, and low semantic richness, then sampled 140 from each group using a fixed random seed for full reproducibility. After manual filtering to remove controlled-access highways, isolated corridors, and heavily occluded scenes, the final benchmark contained 315 images. Each was manually assigned to one of three road contexts: Urban (231 images), Residential (42), or Rural (42). This context label became a critical input for the framework — because the same semantic feature can carry entirely different safety implications depending on where it appears.

CARSA itself is refreshingly transparent. Instead of a black-box neural network, it uses a rule-based scoring system built on three infrastructure factors: pedestrian support (sidewalks, crosswalks, curbs), traffic control (signs, lights, lane markings), and physical protection (barriers, poles). Each factor combines binary presence — does the element exist at all — with normalized pixel coverage — how visually prominent it is. Presence receives the heavier weight, reflecting the insight that humans judge safety primarily by whether protective infrastructure exists, with coverage adding nuance about its visibility.

Here is where the context-awareness kicks in. The three factors are weighted differently for each road type. Urban scenes prioritize pedestrian support at 0.40 and traffic control at 0.35, because dense interaction between vehicles and vulnerable road users makes these elements critical. Residential scenes shift emphasis toward pedestrian support (0.35) and physical protection (0.40), reflecting quieter streets where separation matters more than complex signaling. Rural scenes flip the hierarchy entirely: physical protection dominates at 0.50, traffic guidance at 0.35, and pedestrian support drops to just 0.15 — because dedicated sidewalks are simply not expected on country roads.

Crucially, the researchers froze every parameter before any human evaluation began. No rule, weight, or threshold was tuned using expert labels — a methodological discipline that prevents information leakage and makes the evaluation genuinely independent. The expert panel then rated all 315 images independently, using a three-level scale of Low, Medium, and High safety. The panel included a transportation engineer, a driving instructor, and a computer vision researcher, deliberately combining infrastructure expertise, practical road experience, and visual interpretation skills.

The reliability of this human consensus was remarkable. The Intraclass Correlation Coefficient reached 0.953, Cronbach’s Alpha hit 0.954, and Fleiss’ Kappa came in at 0.805 — indicating almost perfect agreement among the three experts. Complete agreement on all three ratings occurred for 85.08 percent of images. This stability matters enormously: without a trustworthy human reference, no AI framework could be meaningfully evaluated against it.

When CARSA’s predictions were compared against this expert consensus, the results were striking. The framework achieved 92.38 percent accuracy, a weighted F1 score of 92.61 percent, Cohen’s Kappa of 0.853, and Quadratic Weighted Kappa of 0.891. Most disagreements were confined to adjacent safety categories — meaning when CARSA got something wrong, it was usually off by one level rather than wildly misjudging the scene. Only a single image out of 315 showed the maximum possible disagreement.

The comparison with baseline models tells perhaps the most interesting story. A non-contextual semantic ablation — the same factors but with equal weights and global thresholds — managed only 75.87 percent accuracy. That 16.51 percentage point gap, statistically significant after Holm correction, proves that context-specific weighting is not a nice-to-have but the core of what makes CARSA work. Meanwhile, context-aware CLIP zero-shot achieved just 19.68 percent accuracy, Qwen2.5-VL zero-shot 20.32 percent, and Qwen2.5-VL few-shot 29.84 percent. These general-purpose vision-language models, despite their impressive capabilities in other domains, simply could not separate the three safety levels without domain-specific adaptation.

The researchers emphasize that these VLM results should not be read as a general failure of vision-language models. Rather, they highlight how challenging this specific three-level road-infrastructure-safety task remains without specialized training or alignment. CARSA is positioned not as a replacement for these models but as a complementary, interpretable baseline — and potentially a structured component of future hybrid systems that combine transparent contextual reasoning with data-driven adaptation.

Performance varied meaningfully across road types. Urban scenes, rich in semantic information, achieved 96.54 percent accuracy. Rural scenes, despite fewer explicit safety elements, still reached 90.48 percent. Residential environments proved hardest, at 71.43 percent — and the framework showed a consistent conservative bias there, underestimating safety more often than overestimating it. The researchers suggest that residential safety judgments may depend on contextual factors not fully captured by visible infrastructure alone: neighborhood familiarity, tranquility, perceived driver behavior, and environmental quality.

The error analysis revealed something equally fascinating. Framework mistakes were not randomly distributed. Images misclassified by CARSA had an average expert disagreement score of 0.1571, compared to just 0.0632 for correctly classified scenes — a statistically significant difference confirmed by a Mann-Whitney U test with a p-value of 0.0086. Full human agreement occurred in 86.60 percent of correctly classified images but only 66.67 percent of misclassified ones. In other words, when CARSA failed, it often failed on scenes that were inherently ambiguous even for human experts.

A predefined parameter-sensitivity analysis tested 64 alternative configurations around the frozen baseline, perturbing factor weights, decision thresholds, presence-coverage fusion, and coverage normalization. The framework proved generally stable under moderate changes, with accuracy ranging from 87.94 to 93.65 percent across factor-weight perturbations and label stability between 94.92 and 100 percent. No alternative configuration produced a two-level categorical change relative to the frozen baseline — all changed predictions moved only between adjacent safety levels. The most influential components turned out to be the presence-coverage balance and the Urban Medium-High decision boundary.

The study’s most severe failure case is instructive: a residential scene that all three experts unanimously rated as High safety, but which CARSA classified as Low safety — the only maximum-distance disagreement in the entire dataset. Several other large disagreements occurred in images where expert agreement was complete, suggesting that some failures reflect genuine limitations of the current semantic representation rather than human uncertainty. The researchers acknowledge that factors beyond visible infrastructure — familiarity, perceived driver behavior, personal security — may explain some of these gaps.

Beyond the framework itself, the released benchmark represents a significant contribution to the field. It contains 315 contextually categorized street-level images, 945 independent expert ratings, consensus labels, semantic metadata, and complete evaluation outputs. The researchers have released the full rule specification, the complete lists of retained and excluded images, and the source code through the Data and Code Availability section. This transparency enables independent verification of the entire dataset construction procedure and supports full experimental reproducibility.

The implications reach beyond academic benchmarking. As cities worldwide invest in smart infrastructure and autonomous vehicle technology, understanding how humans perceive road safety becomes increasingly critical. Systems that can approximate expert judgment while remaining fully explainable — where every score can be traced back to specific semantic features and contextual rules — offer a path toward trustworthy AI in transportation. CARSA demonstrates that sometimes the most powerful approach is not the most complex one, but the one that most faithfully mirrors how humans actually think.

The researchers are careful to note the study’s limitations. The benchmark comprises 315 images and three experts — an initial reproducible resource rather than a comprehensive representation of all geographic and professional contexts. Performance metrics quantify agreement on the current Mapillary Vistas-derived benchmark and do not establish cross-dataset or cross-geographic generalization. The evaluation assumes correct road-context assignment and reliable semantic inputs; a future end-to-end deployment with automatically predicted masks would introduce upstream perception errors that require separate validation. Future work should evaluate the unchanged framework on independent expert-annotated datasets, examine predicted semantic masks and uncertain context assignments, and explore hybrid approaches combining transparent contextual reasoning with data-driven adaptation.

Still, the core achievement stands: a deterministic, fully auditable framework that approximates multidisciplinary expert judgment with over 92 percent accuracy, outperforming both its own non-contextual ablation and several state-of-the-art vision-language models. In a field increasingly dominated by opaque deep learning systems, CARSA offers something rare — an AI that shows its work, explains its reasoning, and aligns closely with the humans it aims to serve.

Subject of Research: A context-aware and human-centered framework for road safety assessment using semantic scene understanding

Article Title: A context-aware and human-centered framework for road safety assessment using semantic scene understanding

Article References: A context-aware and human-centered framework for road safety assessment using semantic scene understanding. (n.d.). https://doi.org/10.3389/ffutr.2026.1920858

Image Credits: AI Generated

DOI: 10.3389/ffutr.2026.1920858

Keywords: context-aware, human-centered, framework, road, safety, assessment, semantic, scene, understanding, scientific research