AI That Watches Your Eyes: Real-Time Error Prevention for Gaze-Based VR Interaction

Gaze-based interaction has long been celebrated as one of the most intuitive and potentially transformative input modalities for immersive environments. By simply looking at an object, users can select, activate, or manipulate it without ever reaching for a controller, keyboard, or touchscreen. This hands-free quality makes gaze interaction especially attractive for virtual reality (VR), augmented reality (AR), and assistive technologies, where traditional input devices may be impractical or entirely unavailable. Yet this elegance conceals a stubborn technical weakness that has frustrated researchers for decades: the tendency of the system to register unintended gaze behaviour as deliberate input. Known as the Midas touch problem, this phenomenon means that every accidental glance, every wandering fixation, and every moment of visual uncertainty can trigger an action the user never intended. In a VR game, this might cost a few points; in a safety-critical or collaborative scenario, the consequences could be far more serious.

Now a team of researchers at the ZEISS Vision Science Lab at the University of Tübingen and Carl Zeiss Vision International GmbH has developed and rigorously tested a real-time Error Prevention System (EPS) that tackles the Midas touch problem head-on. Rather than relying on predefined error categories or explicit confirmation steps, the system treats selection errors as anomalies within the continuous temporal gaze signal. The approach, described in a study published in Frontiers in Virtual Reality, is grounded in a simple but powerful observation: correct selections are characterized by relatively stable and consistent gaze patterns, whereas erroneous selections often involve increased uncertainty, corrective movements, or unstable fixations. By learning what typical correct selections look like, the system can flag deviations from this learned representation as potential errors before they cause task-relevant consequences.

The technical heart of the EPS is a Temporal Convolutional Network Autoencoder (TCNAE), a model architecture that has already proven effective for anomaly detection in time series data. An autoencoder is a neural network trained to reconstruct its own input: it learns a compressed representation of typical patterns and then attempts to rebuild the original signal from that representation. Crucially, the TCNAE was trained exclusively on gaze dynamics associated with correct selections. Because anomalous selections were never part of the training data, the network should struggle to reconstruct them, resulting in a higher reconstruction error. This error serves as a measure of selection validity: when the reconstruction error exceeds a fixed threshold, the selection is classified as anomalous and the action is interrupted before completion. The threshold was set at the 95th percentile of reconstruction errors obtained from correct selections in the training set, a compromise between detecting anomalous selections and limiting false rejection of correct ones.

The input representation reflects careful engineering choices informed by the realities of real-time eye tracking. Rather than using angular distances between consecutive gaze vectors, which are sensitive to variations in temporal spacing caused by dropped frames or irregular sampling rates, the researchers used angular velocity, which normalizes the angular change by the time difference between samples. The network operates on the last 37 gaze vectors captured in a temporal window centered around the selection event. At a sampling rate of 90 Hz, this window spans approximately 200 milliseconds before and after the selection. The post-selection interval was deliberately kept short: human corrective responses typically occur with a delay of around 300 milliseconds, so using a shorter interval allows the system to respond earlier without relying on explicit user correction. Including gaze data before the selection provides additional context, ensuring the model captures the transition into the selection event.

Training the system required substantial data from a previous experiment conducted in the same virtual environment. The researchers captured all correct selection events from 52 participants, collecting 4,842 correct selections for Dwell Time, 4,725 for Head and Gaze, and 4,926 for Nod. Hyperparameters were determined through five-fold cross-validation performed at the participant level, meaning all samples from a given participant were assigned to the same fold to prevent overlap of participant-specific gaze behaviour between training and validation data. Cross-validation was performed separately for each interaction method, and results were aggregated across methods to find the hyperparameter configuration achieving the best overall performance. A separate network was trained for each interaction method to account for differences in gaze behaviour across techniques. The network was trained for 2,000 epochs with a batch size of 4,000 using the Adam optimizer and Mean Squared Error as the loss function.

To evaluate the EPS in practice, the researchers integrated it into a VR visual search game and recruited 37 participants (26 women and 11 men, mean age 26.14 years) from the University of Tübingen and the surrounding area. Most reported little or no prior experience with eye tracking or VR. The game required participants to identify a target among flying robots positioned in a square-shaped virtual room. At the beginning of each round, robots moved to one of approximately 18 predefined positions arranged within a 180-degree field in front of the user. Once a robot reached its position, a sphere appeared displaying a character: the target was indicated by the letter C, while distractors were indicated by O. Destroying the correct target earned between 5 and 20 points depending on response time, whereas destroying a distractor resulted in a 21-point penalty. Each round lasted 30 seconds, and the scoring scheme was designed to encourage both accuracy and speed under time pressure.

Three gaze-based selection methods were compared, each evaluated with and without the EPS. Dwell Time allowed participants to select a target by fixating on it for a specified duration, set to 0.4 seconds. Gaze and Head required both gaze direction and head orientation to remain aligned with the target for 0.3 seconds. Nod involved a gaze fixation followed by a nod gesture to confirm the selection. When the EPS was active, selections were validated using the TCNAE while the action was being executed and before task-relevant consequences such as target destruction and score updates occurred. Model inference required less than 1 millisecond, negligible compared to the approximately 200-millisecond post-trigger acquisition period needed to complete the input window. If a selection was classified as anomalous, the action was interrupted before completion. The experiment used an HTC Vive Pro Eye head-mounted display with a built-in Tobii eye tracker, and gaze data were accessed once per rendered frame at approximately 90 Hz.

The results are striking. With the EPS active, participants made markedly fewer incorrect selections across all three methods. For Dwell Time, the median number of incorrect selections dropped from 22 without EPS to 1 with EPS, a reduction of up to 95 percent. A similar effect was observed for Head and Gaze, with medians falling from 8 to 1. For Nod, the reduction was statistically significant but the median remained 0 in both conditions, indicating that the benefit was driven by the few participants who made errors rather than by a broad shift in behaviour. Participants also gained substantially more points when the EPS was active: the median score for Dwell Time increased from 435 to 925, and for Head and Gaze from 745 to 819. No significant difference was observed for Nod, where medians were nearly identical at 787 without EPS and 789 with EPS. The EPS itself achieved strong classification performance: for Dwell Time, 79.4 percent of correct selections were classified correctly and 92.8 percent of incorrect selections were classified as incorrect; for Gaze and Head, the corresponding figures were 73.0 and 82.1 percent; for Nod, 90.8 and 60.7 percent.

Subjective evaluations were equally revealing. Participants rated the system positively overall, reporting improved confidence and usability. Of the 37 participants, 31 said the EPS was helpful, only 1 rated it as not helpful, and 5 were not sure. Gaze and Head was the most frequently preferred interaction method, chosen by 16 participants, followed by Nod with 10 and Dwell Time with 11. Questionnaire analysis revealed significant differences between interaction methods for helpfulness, trust, confidence boost, and preference, with the Nod method receiving significantly lower ratings than Dwell Time and Head and Gaze across these dimensions. No significant differences were found for frustration or disruption, where median ratings were similar and generally low. Notably, participants’ ability to recognize whether the EPS was active was only slightly above chance level (50 percent) across methods, suggesting the system worked largely in the background without disrupting interaction. Interestingly, even participants with comparatively low accuracy scores stated they appreciated the system, suggesting that occasional false rejections were not perceived as disruptive, since rejected targets could be immediately reselected while visual attention remained focused on them.

The findings demonstrate that anomaly-based error prevention can effectively reduce incorrect gaze-based selections in real time without disrupting interaction, addressing a challenge that has persisted since the earliest days of eye movement-based interaction. However, the variability across interaction methods and users highlights that a uniform approach may not be sufficient. System performance varied across participants: while it was nearly perfect for some, it was less reliable for others, suggesting that individual gaze behaviour plays an important role in error detection. The authors note that previous research has shown gaze behaviour is highly individual and context-dependent, pointing toward the need for adaptive approaches, such as dynamic adjustment of thresholds or detection strategies to better accommodate individual characteristics and interaction styles. The study also has limitations: the findings relate specifically to a visual search task in VR and may not transfer directly to other applications, and the system was trained to distinguish selection correctness rather than directly infer user intention. Future work should validate the system in a broader range of environments, including AR and assistive technologies, and incorporate independent measures of intention. Nevertheless, the results represent a significant step toward gaze-based interfaces that are not only intuitive but also trustworthy, bringing the promise of truly hands-free interaction in VR, AR, and beyond closer to reality.

Subject of Research: Real-time anomaly detection for preventing unintended gaze-based selections in virtual reality

Article Title: A real-time error prevention system for gaze-based interaction in virtual reality based on anomaly detection

Article References: Severitt, B. R., Sauer, Y., Castner, N., & Wahl, S. (2026). A real-time error prevention system for gaze-based interaction in virtual reality based on anomaly detection. Frontiers in Virtual Reality, 7, Article 1918264. https://doi.org/10.3389/frvir.2026.1918264

Image Credits: AI Generated

DOI: 10.3389/frvir.2026.1918264

Keywords: gaze-based interaction, virtual reality, anomaly detection, Midas touch problem, error prevention, eye tracking, temporal convolutional network, autoencoder, dwell time, human-computer interaction, assistive technology, user experience