Eye tracking reveals how the brain resolves spoken references

A face-to-face experiment in which one person gives instructions to move objects while another follows them has produced a striking map of how language, vision and attention interact in real time. Researchers combined mobile electroencephalography, or EEG, with eye tracking and audio recording as German-speaking pairs communicated around a shelf of objects. The results show that listeners can resolve potentially ambiguous instructions by looking directly to the most informative object, while the brain’s response to the spoken object name reflects how much attention must be redirected to select it. The study is a proof of concept for analysing neural activity during ordinary, mobile interaction rather than while participants stare at a computer screen.

The experimental arrangement was a simplified version of the “director task,” a well-known paradigm in which one participant knows which object their partner should move, even when the partner can see several matching objects. A shelf with 16 compartments stood between the two people, and some compartments were hidden from the director’s view. Six objects were placed on the shelf: two were unique, while the other four formed two pairs of identical items, including candles, flowers, vases and cups. When the director requested one member of a pair, the partner could identify it by noticing whether that object was hidden from the director. All object names were feminine, bisyllabic German nouns, preventing grammatical gender from providing an unintended clue to the intended referent.

The researchers analysed 26 native German speakers; seven further participants were excluded because of technical problems, including disconnected eye-tracking cables, human error and loss of wireless connectivity. The analysed participants were 14 women and 12 men, with a mean age of 24.94 years. Each person acted as the listener, or “matcher,” while a trained confederate played the director. Participants wore a 128-channel mobile EEG system and Pupil Core eye-tracking glasses, which recorded pupil position at a nominal 120 hertz alongside a 30-hertz view of the scene. Four fiducial markers around the shelf allowed the software to track all 16 compartments even as the listener moved their head.

Solving the problem in the physical world required an unusual analytical workflow. EEG samples brain voltage at hundreds of times per second, eye trackers sample gaze at a different rate, and audio equipment operates at yet another timescale. The team used the open-source LabStreamingLayer protocol to timestamp and synchronize the three streams, with the recording computer serving as the master clock. The approach allows researchers to ask not merely whether a neural response followed a word, but whether the same response was stronger when the listener happened to be looking at the relevant object at that moment.

Natural interaction also creates movement artifacts, overlapping brain responses and a lack of strict experimental timing. To clean the EEG data, the team combined artifact subspace reconstruction, which detects transient high-amplitude disturbances and reconstructs brain activity from cleaner portions of a recording, with independent component analysis, which separates statistically independent neural and non-neural sources. Their central statistical innovation was regression-based event-related potentials, or rERPs. Rather than averaging short epochs and assuming that responses do not overlap, this method models continuous EEG as the combined result of responses to spoken words, fixations and other events. Linear deconvolution then estimates the response attributable to each event while accounting for the others.

The results overturned one of the study team’s predictions. Because a hidden competitor reveals information about what the director can see, the researchers expected listeners to inspect ambiguous objects. Instead, the competitors received significantly less visual attention than other non-target objects. Between 1.4 and 3.4 seconds after the critical noun, listeners looked at the hidden competitor less often than at the other objects on the shelf. By contrast, they looked more often at the target when it had a hidden competitor than when it was a unique object, with a significant difference between 1.1 and 1.9 seconds after the noun. The target was both the object to move and the one that disclosed whether the instruction was ambiguous.

This strategy appears to be an example of cognitive offloading. Rather than retaining a detailed mental map of the entire shelf, listeners used the visual scene to obtain the one piece of information they needed. Hidden competitors remained in place throughout a 10-to-15-minute block, whereas targets changed position repeatedly, so little could be gained by looking at the competitors after their locations had been learned. Fixation-related brain potentials supported this interpretation. When a target was viewed during or after its name was spoken, the visual and linguistic information could be checked against one another, producing stronger and earlier neural effects than when a target was found before its name.

Responses time-locked to the spoken nouns also differed between the paired and unique conditions. In the N400/P300 interval, conventionally extending from 300 to 500 milliseconds after word onset, paired targets produced a pronounced positivity over posterior and occipital electrodes, peaking at about 500 milliseconds. The polarity reversed over frontal and prefrontal regions, and a later positivity appeared between roughly 600 and 850 milliseconds. The authors interpret the posterior response as a P300-like signal of attentional reorientation: hearing “candle” required listeners to detect that the referent was not unique, redirect attention and initiate a visual search. A unique target behaved more like an expected item and required less reorientation.

Fixation timing carried much of the interpretive information. In the paired condition, later target fixations were associated with a stronger posterior positivity, especially from 300 to 500 milliseconds after the noun. If listeners had already found the target before hearing its name, they could prepare for it; if they found it during the noun, visual and linguistic evidence arrived together; if they searched only after the word ended, target detection was delayed and required more attentional effort. The study’s resampling analysis showed that the competitor effect in gaze was highly robust, while the key interaction between fixation timing and condition in the fixation-related potentials stabilized at a median sample size of 16. The language-related effects were less secure: the 400-to-500-millisecond N400 effect was the most stable component, whereas several other time windows and higher-order interactions did not stabilize across reduced samples.

The authors are careful not to present the study as a definitive account of language-specific brain mechanisms. Its strongest contribution is methodological: it demonstrates that synchronized mobile EEG and eye tracking can capture language processing during a real, face-to-face exchange, even while participants move and produce movement-related noise. The workflow also preserves the complementarity of the two signals. Gaze reveals what a participant is inspecting and when, while EEG reveals how the brain responds to a spoken word or a fixation. Used together, they can distinguish whether a neural difference reflects the presence of ambiguity, the timing of visual search, or simply the number of looks at an object. All analysis scripts and two example datasets are openly available, while protected participant data may be shared on request.

Future versions could extend the approach in several directions. Recording the director’s gaze, facial expression and prosody would allow both sides of the exchange to be analysed, potentially revealing joint attention as speech is planned and interpreted. More varied utterances would also test whether the workflow survives less constrained language than the short noun phrases used here. The same regression and overlap-correction framework can also be adapted from voltage time courses to continuous EEG-derived measures such as band-limited power, opening the door to oscillatory analyses of communication. For now, the results show that even a simple instruction such as moving a candle can recruit a dynamic loop of prediction, visual search and neural reorientation—and that measuring the eyes alongside the brain can reveal that loop in remarkable detail.

Subject of Research: How eye movements and brain activity jointly support referential selection during spoken interaction

Article Title: Combined eye tracking and electroencephalography during referential selection in dyadic interaction

Article References: Brilmayer, I., Georgis, P., & Schumacher, P. B. (2026). Combined eye tracking and electroencephalography during referential selection in dyadic interaction. Behavior Research Methods, 58(11), Article 326. https://doi.org/10.3758/s13428-026-03172-0

Image Credits: AI Generated

DOI: 10.3758/s13428-026-03172-0

Keywords: mobile EEG, eye tracking, referential selection, language processing, N400, P300, dyadic interaction, naturalistic neuroscience, regression ERP, cognitive offloading, Combined, tracking