Artificial intelligence is learning to “look” less—and perform better. A research team in Japan has developed a multimodal AI system that listens to a video before deciding which moments deserve visual attention. By ignoring most of the footage and concentrating on a small number of informative intervals, the model reduced the time needed to analyze a two-minute video from 52 seconds to just 18 seconds, while still achieving state-of-the-art accuracy in job-interview assessment.
The system was developed by Professor Shogo Okada and doctoral researcher Hung Le at the Japan Advanced Institute of Science and Technology (JAIST). Their model, called EMF-dVAE, is designed to address one of the central problems facing modern artificial intelligence: the enormous computational cost of processing multiple forms of information at once. Today’s AI systems increasingly work with combinations of video, audio, language, and images, but conventional approaches often examine every frame, even when most contain little useful information. This not only wastes processing power but can also introduce visual noise that makes predictions less reliable.
The researchers designed EMF-dVAE around a principle that is familiar to humans: attention should be selective. During a conversation, people do not stare continuously at another person’s face or monitor every movement with equal intensity. Changes in tone, pauses, emphasis, or other sounds may first signal that something important is happening. Only then does visual attention shift toward the speaker’s expression, gestures, or posture. EMF-dVAE imitates this sequence by using audio to identify moments that may contain valuable visual information.
The model combines two major components: a discrete variational autoencoder, or dVAE, and a multimodal fusion network. A variational autoencoder is a type of neural network that learns to represent complex data in a compressed form and reconstruct it. In this system, the dVAE is trained with partially corrupted visual data. Audio information determines which portions of the video should be masked, and the network must then reconstruct the missing visual content. By learning which visual segments are easiest or most important to recover in relation to the audio, the system develops an internal sense of which moments are likely to matter.
This training strategy allows the model to learn visual relevance without treating every frame equally. Once training is complete, the dVAE can select only a limited number of visual segments during the analysis of a new video. These selected features are then passed to the multimodal fusion network, where they are combined with audio and language data, including the spoken transcript. The fusion network uses all three information streams to produce the final prediction, while the majority of video frames are excluded before expensive visual processing takes place.
The team evaluated the model using the ETS-Interview dataset, which contains 1,891 two-minute job-interview videos recorded from 260 participants. Such videos require AI systems to interpret several layers of human communication at once, including spoken content, vocal delivery, facial behavior, body movement, and other visual signals. EMF-dVAE used only 15.42 percent of the available visual features, effectively discarding nearly 85 percent of the visual data. Despite this drastic reduction, it achieved state-of-the-art performance on the dataset.
The efficiency gains were equally striking. Processing time fell by approximately 65 percent, from 52 seconds per video to 18 seconds. The result suggests that reducing the amount of data an AI system sees does not necessarily make it weaker. In some cases, removing redundant frames may improve performance because irrelevant images can obscure the signals that matter most. The approach also reduces the computational resources required for multimodal analysis, potentially lowering energy consumption and making advanced AI more practical on less powerful hardware.
The researchers believe the technology could support a new generation of real-time communication tools. AI-powered interview coaches might analyze only the moments in which a candidate’s vocal delivery and visual behavior become especially informative, then provide feedback at a cost closer to that of ordinary software. Similar systems could assist with communication training, tutoring, accessibility tools, and robots designed to interact naturally with people. Because the model adjusts how much video it processes for each clip, it could also make multimodal applications more responsive on everyday devices.
Professor Okada argues that selective attention will become increasingly important as video becomes the dominant form of digital data. Systems that attempt to process every frame of every video may eventually become too expensive, slow, and environmentally demanding to scale. By budgeting its attention, AI could instead devote computational power to moments that carry the greatest meaning. The researchers envision that within the next decade, this principle could help make multimodal assistants, interview coaches, educational systems, and communication-support robots faster, more affordable, and more sustainable. The study’s findings were made available online on July 11, 2026, and the full article is scheduled for publication in Information Fusion on January 1, 2027.
Subject of Research: Computational simulation/modeling
Article Title: Audio-guided visual selection for efficient multimodal fusion via a discrete variational autoencoder
News Publication Date: July 11, 2026
Web References: https://doi.org/10.1016/j.inffus.2026.104613
References: 10.1016/j.inffus.2026.104613
Image Credits: Professor Shogo Okada from JAIST, Japan
Keywords
Artificial intelligence, multimodal AI, video analysis, audio-guided visual selection, discrete variational autoencoder, machine learning, information fusion, computational efficiency, job-interview analysis, sustainable AI
