Depth perception drives sound visualization in virtual reality experiences

In a finding that challenges long-held assumptions about how our brains merge sight and sound, researchers at the University of Utah have shown that the famous “ventriloquist effect” survives the leap into virtual reality, but with a surprising twist. When people wearing immersive headsets tried to align real-world sounds with virtual objects, their brains pulled the sounds toward the objects just as previous laboratory work predicted. Yet the direction of that pull was reversed relative to classic real-world studies, and the culprit appears to be one of virtual reality’s best-known flaws: the systematic underestimation of visual distance. The study, published in the journal Attention, Perception, & Psychophysics, suggests that the fundamental rules of multisensory integration are robust enough to transfer into synthetic worlds, but that the distorted spatial perception those worlds induce can flip the sign of established perceptual effects.

The phenomenon at the heart of the research is known as visual capture, or more specifically in the depth dimension, the proximity image effect. First described in the acoustics literature in the late 1960s, the proximity image effect refers to the tendency of listeners to perceive a sound as originating closer to a plausible visual source than it actually is. When a ventriloquist’s puppet’s mouth moves, the audience hears the voice coming from the puppet, not the performer. In laboratory settings, when a sound is emitted slightly behind or in front of a visible object, listeners systematically misjudge its location, biasing it toward the object. Decades of work, including influential Bayesian models of cue integration, have explained this as the brain weighing visual information more heavily than auditory information for spatial localization, particularly along the depth axis where hearing is comparatively imprecise. Vision dominates because it is typically the more reliable sense for spatial tasks, and the brain behaves like a near-optimal statistical integrator, combining sensory estimates weighted by their respective reliabilities.

What remained unclear, however, was whether these principles, derived mostly from experiments in real rooms with real objects, generalize to virtual environments. Virtual reality is not a neutral replica of the physical world. Decades of research have documented a persistent “distance compression” effect in head-mounted displays: people consistently judge objects in VR as closer, often around 74 to 90 percent of their actual distance, even in modern high-resolution consumer headsets. If the visual representation of space is systematically shrunk, the question becomes how the auditory system, which in these experiments received real, physically accurate sound, interacts with a visual system operating on distorted coordinates. The Utah team, led by Maggie K. McCracken together with Macy E. Rowland, Hunter C. Finney, Jeanine K. Stefanucci, and Sarah H. Creem-Regehr, designed two experiments to answer exactly this question.

The experimental logic was elegant. Participants wore an immersive head-mounted display in a real laboratory room while hearing sounds from an actual, physical sound source positioned at various depths. At the same time, they saw a virtual object rendered at a specified depth in the virtual scene. Their task was a perceptual matching judgment: they indicated when they perceived the real-world sound source to be aligned in depth with the virtual object. If no visual capture occurred, their judgments should track the true physical co-location of sound and virtual object. If visual capture occurred, their points of subjective alignment should be biased toward the visible object, and the magnitude and direction of that bias would reveal how the brain was combining the two spatial estimates. Crucially, this setup created a genuine cross-reality conflict: the ears received physically veridical information about the real world, while the eyes received rendered information about a virtual object occupying a different depth position.

The results of the first experiment delivered a double surprise. Visual capture did reliably occur, confirming that the multisensory machinery at work in physical rooms also operates when one of the signals belongs to a synthetic environment. But the asymmetry of the effect ran opposite to what real-world studies had established. In classic experiments, capture was stronger when the sound originated behind the visual object, presumably because listeners are more uncertain about sounds located behind opaque surfaces and therefore lean more heavily on vision. In the Utah study, capture was stronger when the sound originated in front of the virtual object rather than behind it. At first glance, this reversal looked like a failure of the classical model, a suggestion that the ventriloquist effect might not behave the same way in VR at all.

The second experiment resolved the puzzle. The team replicated the reversed asymmetry pattern, but they also added an independent measurement of the participants’ visual distance perception using a blind-walking task, a gold-standard procedure in which participants view a target and then walk to it without vision, allowing researchers to infer perceived distance from walked distance. The measurements confirmed what the VR literature had long predicted: participants significantly underestimated the visual distances of virtual objects. When the researchers reanalyzed the audiovisual alignment judgments not relative to the physical distance of the virtual objects, but relative to each participant’s perceived, compressed visual distance, the expected real-world symmetry reemerged. In other words, the brain’s integration rule had not changed; the coordinate system had. Once the analysis was anchored to what participants actually perceived visually rather than where the virtual object “really” was in the rendered scene, the classic pattern of capture fell back into place.

This reframing carries significant theoretical weight. It implies that multisensory integration operates on perceived space, not physical space. The brain does not have access to the ground-truth coordinates programmed into the virtual environment; it only has access to the noisy perceptual estimates generated by the visual and auditory systems. If the visual estimate is compressed, then the point at which sound and vision appear co-located shifts accordingly, and the pattern of capture shifts with it. The findings thus demonstrate that core principles of multisensory integration, including visual dominance in depth and the statistical weighting of cues by reliability, are preserved in virtual reality, but that VR-specific biases can alter, and even reverse, the direction of well-established effects when measured against physical reality. This is a powerful demonstration that the same computational rules apply across real and virtual worlds, provided researchers account for how those worlds are actually perceived.

The methodological implications are substantial for anyone building or studying virtual environments. Developers of spatial audio for VR often assume that rendering sound at a physically correct position relative to a visually rendered object will produce the intended percept. This study shows that the perceived relationship between a real sound and a virtual object can deviate in systematic, predictable ways from the physical arrangement, and that these deviations are driven by the visual system’s compressed representation of depth. Applications that depend on precise audiovisual alignment, such as virtual training simulations, spatial audio design for games and cinematic VR, accessibility tools that use sound to guide attention to visual targets, and clinical or research paradigms involving multisensory integration, may all need to compensate for distance compression when placing sounds relative to visual content. Conversely, the effect could be exploited deliberately: if visual capture reliably pulls perceived sound location toward virtual objects, designers could use visual anchors to sharpen the perceived localization of audio without changing the audio itself.

The work also contributes to a deeper scientific conversation about why distance compression occurs in head-mounted displays and how it interacts with other perceptual processes. Hypotheses have ranged from limited field of view and display resolution to the absence of accurate accommodative and vergence cues, the influence of the virtual environment’s layout, and the lack of natural body-based scaling information. By showing that distance compression propagates into the domain of multisensory perception, the study adds a new consequence to the list: the distortion does not merely make virtual rooms feel smaller, it reshapes how the brain binds together information from different senses within those rooms. That makes distance perception in VR not just a curiosity of display technology but a foundational parameter that conditions virtually every spatial judgment a user makes.

The research, conducted at the University of Utah’s Department of Psychology and approved by the university’s Institutional Review Board, was supported in part by a National Science Foundation Graduate Research Fellowship. Trial-level data and analysis scripts for both experiments have been made publicly available on the Open Science Framework, reflecting a growing commitment to transparency in perception science. As head-mounted displays become everyday consumer technology and spatial audio becomes a standard feature of digital experiences, understanding how the brain reconciles real sound with virtual sight will matter far beyond the laboratory. This study offers both a caution, that virtual worlds distort space in ways that silently reshape multisensory perception, and a reassurance, that the brain’s integration rules remain coherent, lawful, and ultimately predictable even in worlds that do not physically exist.