Synthetic city crowds teach autonomous cars the art of predicting pedestrian crossings

Every autonomous vehicle must answer one deceptively simple question thousands of times per drive: is that pedestrian about to step into the road? Getting it wrong translates directly into collision risk, yet the datasets used to train crossing-prediction models are riddled with blind spots. Crossing events form a small minority of the frames captured by dashboard cameras, and the behaviours that matter most for safety—jaywalking, sudden dashes, mid-road pauses—are precisely the ones that real-world recordings cannot ethically stage. A team of researchers at the Barcelona Computer Vision Center and AGH University of Science and Technology has now released a software pipeline, ARCANE-PedSynth, that manufactures exactly those missing moments inside a simulator, complete with dense behavioural annotations and a controllable crossing rate. The framework, published in the journal SoftwareX, has already generated PedSynth++, a dataset of 533 video clips spanning 177,231 annotated frames, 3,426 unique pedestrians, and eleven weather conditions across four virtual towns.

The motivation stems from a stubborn statistical problem. Established real-world datasets such as JAAD and PIE have been instrumental in advancing pedestrian behaviour prediction, providing behavioural annotations from dashboard cameras, but crossing frames constitute only a minority of observed data. Manual annotation is expensive and prone to inconsistency, particularly for subjective attributes such as crossing intention and attention state. Moreover, real-world recordings cannot ethically stage hazardous edge cases: no ethics board will approve staging a sudden dash into oncoming traffic. Earlier synthetic efforts demonstrated the feasibility of training perception models on simulator output, but CARLA’s built-in AI pedestrian system routes walkers along sidewalks, yielding an observed baseline crossing rate of only 8.9 percent in the team’s release configuration—insufficient for training balanced crossing/non-crossing models.

ARCANE-PedSynth overcomes this baseline with a dual-mode control architecture that switches between AI-controlled sidewalk navigation and direct manual commands specifying crossing trajectories, speeds, pauses, and retreats. The system computes a perpendicular crossing vector by querying CARLA’s waypoint API for the nearest road waypoint and calculating the direction perpendicular to the road’s forward vector. A four-layer crossing-rate engineering strategy then shapes the data: potential crossers are allocated at spawn, behavioural types are assigned, mid-road jaywalkers are injected directly into driving lanes, and groups of two or three pedestrians are formed with synchronised crossing behaviour. In a cumulative ablation confined to a single town and clear noon weather, the native CARLA AI baseline of 3.8 percent rose to 47.3 percent with crosser allocation alone, then climbed to 56.8 percent once all four layers were active. Layers one and three chiefly control the rate, while layers two and four shape behaviour and group composition.

Each pedestrian is driven by a twelve-state behavioural finite state machine whose transitions depend on local traffic context and a fixed character archetype. Five archetypes—business person, casual person, elderly person, young person, and parent with child—span a designed range of walking speeds from 0.6 to 1.8 metres per second, providing controlled behavioural diversity rather than a faithful simulation of individual human cognition. The decision to step onto the road follows a gap-acceptance model based on time-to-collision: for each pedestrian position and each vehicle within sensor range, the framework computes the projected closing speed, and a crossing is admissible when the minimum time-to-collision across vehicles exceeds the time required to traverse the road plus a safety margin. Once a pedestrian has entered the road, vehicle proximity is re-evaluated at three lookahead points along the crossing vector; if any vehicle falls within a three-metre safety radius, the agent halts and may pause mid-road or, with fifteen percent probability while still in the first forty percent of the crossing path, reverse direction and return to its starting kerb—producing the mid-road reversals and near-misses that are particularly valuable for safety-critical training.

The framework captures three synchronised sensor modalities mounted on the ego vehicle: an RGB camera at driver eye level with 1280×720 resolution, a roof-mounted 64-channel LiDAR generating one million points per second, and a dynamic vision sensor (DVS) event camera co-located with the RGB camera. The LiDAR provides 3D point clouds whose geometric measurements do not depend on scene illumination, since it is an active sensor, though adverse conditions such as rain and fog still degrade returns through scattering and attenuation. The DVS stream is produced by CARLA’s event-camera model, which emits events by thresholding the change in per-pixel log intensity between consecutive rendered frames—a rendering-based approximation whose temporal resolution is bounded by the frame rate rather than the microsecond timing of real hardware. All sensors run in CARLA’s synchronous mode indexed by frame, with RGB and DVS produced at 30 frames per second and the 20-hertz LiDAR associated with the frame in which its sweep completes.

A four-stage visibility gate determines which pedestrians are annotated: distance to ego below seventy metres, positive dot product with the camera forward vector, projected bounding box of at least fifteen by thirty pixels, and bounding box within image bounds. Simulator-derived ground-truth labels read directly from CARLA’s internal state include lane-validated crossing labels, behavioural state, bounding boxes, distance to ego, time-to-event markers following JAAD and PIE conventions, and character archetype. Separately, estimated pose labels—17-keypoint COCO-format 2D keypoints—are produced by AlphaPose from the rendered RGB frames as a post-processing step. These are pseudo-labels rather than simulator ground truth: on 800 matched detections between zero and thirty metres, AlphaPose achieved a mean keypoint confidence of 0.68, degrading across distance bands from 0.74 at zero to ten metres to 0.56 at twenty to thirty metres, while OpenPose fell from 0.34 to zero, producing no usable estimates beyond twenty metres. AlphaPose is therefore the recommended extractor based on this comparison.

The released PedSynth++ dataset demonstrates the pipeline at scale: 533 clips across eleven weather conditions and four CARLA towns, with each clip running approximately ten to fifteen seconds at 30 frames per second and containing five to ten pedestrians alongside five to ten traffic vehicles with varied driving profiles—twenty percent aggressive, sixty percent normal, twenty percent cautious—managed by CARLA’s Traffic Manager. The controllable crossing rate of forty to seventy-five percent is the dataset’s most distinctive characteristic, enabling researchers to generate balanced fifty-fifty splits for standard training, imbalanced distributions matching real-world conditions for evaluation, or crossing-heavy distributions for focused rare-event training. The full dataset occupies 267 gigabytes, with generation taking approximately twelve minutes per clip and a modest peak GPU memory footprint of 3.2 gigabytes, indicating deployment on mid-range hardware rather than high-end systems.

To verify that the generated annotations are internally consistent and learnable, the team evaluated crossing/non-crossing prediction on PedSynth++ against three established models—ST-CrossingPose, PedGraph+, and PedGT—all receiving identical input: each pedestrian’s 17 2D pose keypoints together with the bounding-box centre point. Following JAAD and PIE evaluation protocols, observation windows of eight, sixteen, and thirty frames were tested, with data split at the clip level so that no pedestrian appears in more than one set. PedGT at sixteen frames delivered the best overall trade-off, reaching 86 percent accuracy, 87 percent precision, an F1 score of 89, and an AUC-ROC of 89, leading both baselines on accuracy, precision, and F1. PedGraph+ and ST-CrossingPose improved with longer windows, both peaking at thirty frames, where PedGraph+ matched PedGT’s AUC-ROC of 89. Across all three models, the shortest windows favoured recall over precision. These results confirm in-domain learnability across distinct architectures, though they do not by themselves prove sim-to-real generalisation, which requires domain adaptation techniques and remains the subject of ongoing work.

The framework’s impact extends beyond the dataset itself. The behavioural finite state machine produces behaviours scarce in most existing collections: sudden crossings, jaywalking, distracted walking, mid-road pauses, retreats, and synchronised group crossings. The earlier PedSynth dataset has already demonstrated that combined synthetic-real training outperforms real-only baselines, and the dataset has been employed to reduce manual annotation requirements for pedestrian intention prediction, enabling semi-supervised labelling workflows where synthetic annotations serve as pseudo-labels for real-world data. Docker containerisation ensures consistent CUDA versions and dependency management across machines, and new scenario types or behavioural models can be added by subclassing existing Python classes without modifying the core framework, which is released under the MIT Licence.

The authors are candid about limitations. The generated data remains synthetic and inherits the behavioural constraints of the CARLA simulator: the finite state machine increases crossing diversity but does not fully model pedestrian-pedestrian negotiation, gaze-based communication with drivers, social group dynamics, or culturally dependent crossing norms. The five character archetypes provide parametrically varied crossing profiles rather than reproducing the full complexity of human pedestrian cognition, and real populations exhibit a continuous spectrum of risk tolerance and attention that these discrete profiles approximate rather than replicate. The sidewalk-anchored spawning strategy requires contiguous sidewalk lanes adjacent to driving lanes, excluding highway-dominated maps such as CARLA’s Town04. The RGB, LiDAR, and DVS modalities have not yet been benchmarked for pedestrian crossing prediction, and the synthetic DVS stream does not reproduce physical event-sensor characteristics such as microsecond timing or sensor noise. Sim-to-real transfer remains an active research direction, with future work including parallel generation, LiDAR-assisted 3D pose lifting, and richer pedestrian interactions.

Subject of Research: ARCANE-PedSynth: Synthetic multi-pedestrian datasets with behavioural crossing annotations

Article Title: ARCANE-PedSynth: Synthetic multi-pedestrian datasets with behavioural crossing annotations

Article References: Riaz, M. N., Wielgosz, M., & López Peña, A. M. (2026). ARCANE-PedSynth: Synthetic multi-pedestrian datasets with behavioural crossing annotations. SoftwareX, 36, Article 103115. https://doi.org/10.1016/j.softx.2026.103115

Image Credits: AI Generated

DOI: 10.1016/j.softx.2026.103115

Keywords: ARCANE-PedSynth, Synthetic, multi-pedestrian, datasets, behavioural, crossing, annotations, scientific research