Reaching Out Warps What You Hear and See: Action Reshapes Multisensory Perception

Every time you reach for a coffee cup, your brain quietly reorganizes the way it blends sight and sound. That is the striking implication of a new study published in the journal Attention, Perception, & Psychophysics, in which researchers from Washington State University, Kyushu University, the University of Toronto, and Waseda University show that the simple act of pointing at a target changes how accurately people perceive audiovisual events, and that this perceptual reshaping happens only at the location the action is aimed at, and only while the movement is actually underway. The findings challenge the long-standing assumption that multisensory integration is a largely automatic process that unfolds the same way across the visual field whenever sights and sounds happen to coincide in space and time.

The study, led by Tristan Loria with colleagues including Kanji Tanaka, Joëlle Hajj, Katsumi Watanabe, and Luc Tremblay, exploited one of the most famous tricks in perception science: the sound-induced flash illusion. In its classic form, a single flash of light accompanied by two brief beeps is often perceived as two flashes, an error known as the fission illusion. The complementary version, the fusion illusion, occurs when two flashes paired with a single beep collapse perceptually into one. First described by Shams, Kamitani, and Shimojo in 2000, the illusion demonstrates that audition can literally rewrite what we think we see, and it has since become a standard probe for measuring how the brain combines information across the senses.

For decades, the dominant view held that such cross-modal binding occurs automatically whenever stimuli are spatially and temporally congruent, following principles such as near-optimal Bayesian integration, in which the brain weights each sense according to its reliability. But a growing body of evidence has complicated that picture. Attention, cognitive load, expectation, and even clinical conditions such as glaucoma, migraine, and autism spectrum disorder all modulate how strongly the illusion grips the observer. The new study adds a surprising factor to that list: the body’s own goal-directed movements.

The researchers designed two experiments. In the first, participants performed pointing movements toward visual targets while unimodal visual stimuli, congruent audiovisual pairs, or illusion-inducing audiovisual combinations were presented either at the target location itself or at adjacent non-target locations, both close to and far from the target. After completing each movement, participants reported how many flashes they had seen, giving the team a precise measure of perceptual accuracy for each stimulus type at each location. The logic was straightforward: if multisensory perception is uniform across space, accuracy should not depend on where the stimuli appeared relative to the movement goal.

That is not what happened. The results revealed a clear spatial gradient in perceptual accuracy. Bimodal, congruent audiovisual stimuli were processed more accurately at the target location than at either the close or far non-target locations, indicating that the perceptual benefits of combining sight and sound are concentrated where the action is directed. More intriguing still was the pattern for the fusion illusion. Participants were more susceptible to the illusion, meaning lower perceptual accuracy, at the target location, while accuracy was higher at the nearby non-target positions. In other words, the very spot the brain prioritizes for action became the spot where sound most powerfully distorted vision, collapsing two flashes into one.

The second experiment asked whether this modulation depends on actually moving, or whether merely preparing an action is enough. Participants again pointed at targets while the same battery of stimuli was delivered at the target location, but the researchers compared trials in which the movement was executed against a no-movement condition. The answer was unambiguous: the modulation of multisensory perception at the target location appeared only during movement itself, not when participants remained still. This temporal specificity suggests that the efferent signals and sensorimotor processes accompanying active movement, rather than attentional preparation alone, are what reshape the audiovisual percept.

Perhaps the most technically revealing aspect of the study is a double dissociation hiding in the data. Only the fusion illusion, in which two flashes merge into one, was sensitive to both the spatial gradient and the movement manipulation. The fission illusion, in which one flash splits into two under the influence of two beeps, remained remarkably stable across all locations and both action states. This asymmetry aligns with neurophysiological evidence that the two illusions, despite their mirror-image phenomenology, rely on distinct underlying mechanisms. Prior work using electroencephalography has traced fusion to early cross-modal interactions in auditory and visual cortex, while fission appears to engage different, possibly later, processing stages. The new behavioral results suggest that whatever mechanism goal-directed action recruits, it taps into the circuitry underlying fusion but leaves fission untouched.

The findings resonate with a broader literature on action-centered attention and peripersonal space. Research on selective reaching has shown that attention is organized around the body and its intentions rather than around retinal coordinates, and studies of saccade preparation have demonstrated that attention shifts to the goal of an eye movement before it lands. Earlier work by Tremblay and colleagues had already shown decreased sensitivity to an audiovisual illusion during goal-directed reaching, and a 2020 study by Loria, Tanaka, Watanabe, and Tremblay found that deploying attention to a pointing target modulates audiovisual processing even at non-target locations. The new experiments sharpen that picture by separating spatial proximity from movement execution, showing that the two factors interact: space defines where perception is modulated, and action defines when.

Why would the brain deliberately make itself more vulnerable to an illusion at the location it is reaching toward? One plausible interpretation comes from reliability-weighting models of multisensory perception. Peripheral vision is coarse and unreliable, and when visual information is degraded, the brain leans more heavily on the more precise auditory signal, which is exactly the condition under which fusion illusions flourish. During a goal-directed action, resources are funneled toward the target to guide the hand, and the resulting perceptual economy may amplify the relative weight of sound at that location. The paradoxical outcome is that the behaviorally most relevant spot in space becomes the spot where the senses merge most aggressively, a trade-off between sharpening an action plan and accepting perceptual distortion.

The implications extend well beyond the laboratory. Multisensory integration underpins everything from speech perception and virtual reality design to rehabilitation after stroke and the performance of surgeons, pilots, and athletes working in cluttered, noisy environments. If the fidelity of audiovisual binding shifts dynamically with our actions, then interface designers, clinicians, and human-factors engineers may need to account for a moving, action-dependent perceptual landscape rather than a static one. The authors note that the raw data will be made available upon reasonable request, supporting verification and secondary analysis. For now, the message is vivid and a little unsettling: the world you perceive is not a fixed rendering but a construction that your own movements edit in real time, bending sight around sound precisely where you intend to act.

Subject of Research: Modulation of audiovisual multisensory integration and the sound-induced flash illusion by goal-directed action

Article Title: Goal-directed action modulates audiovisual multisensory perception

Article References: Loria, T., Tanaka, K., Hajj, J., Watanabe, K., & Tremblay, L. (2026). Goal-directed action modulates audiovisual multisensory perception. Attention, Perception, & Psychophysics, 88(7), Article 201. https://doi.org/10.3758/s13414-026-03337-1

Image Credits: AI Generated

DOI: 10.3758/s13414-026-03337-1

Keywords: multisensory integration, sound-induced flash illusion, audiovisual perception, goal-directed action, pointing movements, spatial attention, peripersonal space, fusion illusion, fission illusion, sensorimotor processing, psychophysics, attention