Video platforms are overflowing with footage, but the technology designed to condense that footage into meaningful highlights still faces a deceptively difficult question: what makes one frame more important than another? A new study published in Applied Intelligence introduces a model that attempts to answer that question while selecting video summaries without human annotations. Called ISCRL, short for Interpretable Self-supervised Contrastive Reinforcement Learning, the system combines two rapidly advancing areas of artificial intelligence—self-supervised contrastive learning and reinforcement learning—with visual explanation tools intended to show why particular frames were chosen. The researchers, Jingtao Sun and Jiaxing Wang of Xi’an University of Posts and Telecommunications, report that the approach achieves competitive performance on established video-summarization benchmarks, with especially strong results on the TVSum dataset and in cross-dataset transfer experiments.
Video summarization is often presented as a simple act of choosing the “best” images from a longer recording. In practice, an effective summary must preserve the central meaning of a scene, avoid repeating nearly identical moments, and represent the video with only a small number of frames. These requirements are difficult to satisfy because raw visual features do not always capture high-level semantics. Two frames may look similar at the pixel level while showing different actions, or appear different because of camera motion, lighting, or viewpoint changes while depicting the same event. The challenge becomes even greater in unsupervised settings, where the system receives no manually selected key frames to imitate. Instead, it must learn its own criteria for importance, diversity, and representativeness from the structure of the video itself.
ISCRL begins with a convolutional neural network, or CNN, that extracts visual features from individual frames. These original features preserve information from the video’s initial representation, but they may also contain unwanted sensitivity to changes that should not affect summarization decisions. To improve robustness, the researchers add a self-supervised contrastive learning module based on SimCLR, a widely used framework that teaches a model to recognize different augmented versions of the same image as related. In this setting, frame-level features are transformed into alternate views through data augmentation, and the model is trained to pull representations of corresponding views closer together while pushing representations from unrelated frames farther apart. No labels are required. The goal is to make the learned representation more invariant to superficial changes and more consistent with the underlying visual content.
The contrastive stage is not used as a replacement for the original CNN representation. Instead, ISCRL combines the original and invariant features at the state-representation level of a reinforcement-learning framework. That framework is based on the Deep Self-attention Recurrent Summarization Network with Reinforcement Learning, or DSR-RL. A recurrent component processes the temporal sequence, allowing the model to consider how a frame relates to what came before and after it. Self-attention helps the system assign different importance to moments across the video, rather than relying only on nearby frames. The resulting state describes the evolving context of the video, and a policy network uses that state to decide whether a frame should be included in the final summary.
Reinforcement learning changes the way the selection problem is formulated. Instead of predicting a fixed label for every frame, the model behaves like an agent making a sequence of decisions. Selecting a frame produces consequences that can be evaluated through a reward signal. ISCRL’s most distinctive contribution is a dual-space semantic reward mechanism, which evaluates selected summaries in both the original feature space and the contrastively learned invariant space. In each space, the reward is designed to balance diversity and consistency. Diversity discourages the policy from filling the summary with redundant frames, while consistency encourages the selected frames to remain representative of the source video, including under augmented views. By combining the two spaces, the method attempts to prevent the policy from becoming overly dependent either on raw visual details or on an abstract representation that might discard useful information.
This reward design addresses a long-standing tension in automatic summarization. A summary made entirely of visually different frames may appear diverse but fail to communicate what the video is about. Conversely, a summary that captures one dominant visual pattern may be semantically coherent yet repetitive and incomplete. In ISCRL, the reinforcement-learning agent receives guidance from both requirements at once. The policy is encouraged to identify frames that contribute new information while still remaining connected to the video’s overall content. Because the reward is computed without human annotations, the model can be trained on videos for which no frame-level summary is available. This makes the approach potentially useful for the enormous volumes of online, surveillance, educational, and personal video that cannot be manually labeled.
The study also places unusual emphasis on interpretability. Many video-summarization systems can generate a compact sequence of frames but offer little insight into their decisions. ISCRL combines self-attention with a smoothed Grad-CAM++ technique to produce temporal-spatial explanations. Grad-CAM++ uses gradients flowing through a neural network to identify regions of an image that influence a prediction, while smoothing is intended to make the resulting visual evidence less noisy and more stable. In ISCRL, the explanation process can indicate both when the model considered a frame important and which areas within that frame contributed to the decision. This offers a route toward inspecting whether the model focused on meaningful objects, actions, or scene structures rather than irrelevant background patterns.
Rather than treating interpretability as a purely visual demonstration, the researchers evaluate it through three forms of analysis. Deletion-based faithfulness tests whether removing highlighted evidence changes the model’s decision, providing an indication of whether the explanation points to information the system actually uses. Perturbation-based stability examines whether explanations remain reasonably consistent when the input is changed slightly. Evidence-source informativeness asks whether the highlighted regions and temporal locations contain useful signals for understanding the summary decision. These tests do not prove that an explanation perfectly reveals the model’s internal reasoning, but they provide a more systematic assessment than simply displaying heat maps alongside selected frames.
Experiments were conducted on SumMe and TVSum, two widely used benchmarks for video summarization, with additional evaluations involving canonical, augmented, and transfer settings. The authors report competitive F1-score and rank-correlation results, metrics commonly used to compare automatically generated summaries with human judgments. The strongest findings were observed on TVSum and in cross-dataset transfer evaluation, suggesting that the representation learned through contrastive training may help the policy handle changes in video content and data distribution. Ablation experiments further examined the contributions of the original features, invariant features, and dual-space reward mechanism, while stabilization and complexity analyses addressed training behavior and practical cost. The paper does not portray the model as universally superior, but presents the combined architecture as a way to improve robustness and transparency at the same time.
The significance of ISCRL lies in its attempt to connect three goals that are often pursued separately: learning without labels, selecting summaries through sequential decision-making, and explaining the evidence behind those decisions. Contrastive learning supplies representations designed to be less fragile; reinforcement learning turns summarization into an adaptive selection process; and explanation tools make the resulting policy easier to inspect. The approach could be valuable in applications where users need not only a short video summary but also confidence that the selected moments are meaningful. At the same time, the benchmark results should be interpreted within the limits of current evaluation practices, since human preferences for summaries can vary and feature-space rewards are only indirect measures of semantic understanding. As video collections continue to expand, models that compress footage while exposing the reasoning behind their choices may become an important step toward more trustworthy automated media analysis.
Subject of Research: Interpretable unsupervised video summarization using self-supervised contrastive learning and reinforcement learning
Article Title: An interpretable video summarization model integrating self-supervised contrastive learning and reinforcement learning
Article References: Sun, J., & Wang, J. “An interpretable video summarization model integrating self-supervised contrastive learning and reinforcement learning.” Applied Intelligence, volume 56, article 393, 2026.
Image Credits: AI Generated
DOI: https://doi.org/10.1007/s10489-026-07413-9
Keywords: Video summarization; self-supervised learning; contrastive learning; reinforcement learning; interpretability; key-frame selection; SimCLR; Grad-CAM++
