The research, led by Vandana Pathak of Graphic Era Deemed to be University in Dehradun, together with Manoj Diwakar, Neeraj Kumar Pandey, Sanjay Roka and Prabhishek Singh, addresses a stubborn problem in video surveillance: anomalies are rare, varied and almost impossible to enumerate in advance. A cyclist cutting through a pedestrian zone, a person sprinting the wrong way down a crowded corridor, or an abandoned bag left on a plaza all look nothing alike, which makes supervised classification impractical. The dominant alternative, unsupervised anomaly detection, flips the task on its head. Instead of teaching a model to recognize trouble, it teaches the model to recognize normality so thoroughly that trouble becomes conspicuous by its absence.

The centerpiece of the new framework is a memory-augmented CNN-ConvViT autoencoder. Autoencoders compress an input into a compact latent representation and then attempt to reconstruct it, and when trained exclusively on normal footage they reconstruct familiar patterns well and unfamiliar ones poorly. The reconstruction error then serves as an anomaly score. The difficulty, well documented in prior work, is that a sufficiently powerful autoencoder learns to generalize too much, reconstructing even the anomalies it was never trained on, which erases the very signal the system depends on. Memory modules were introduced to counteract this by storing prototypical features of normal behavior and forcing the encoder to express every input as a combination of those stored prototypes, limiting the network’s ability to improvise.

What distinguishes the new approach is the design of its memory component, called the Temporal-Aware Prototype Memory Module, or TAPMM. Rather than treating memory as a static dictionary of spatial appearances, TAPMM explicitly learns prototypes of normal spatio-temporal behavior, capturing how scenes evolve over time rather than merely how they look in a single frame. This temporal awareness matters because many surveillance anomalies are defined by motion rather than appearance: a person walking calmly through a parking lot is unremarkable, while the same person running through it may warrant attention. By encoding temporal structure into the memory itself, the framework narrows the gap between what the model stores and what constitutes an anomaly in practice.

The architecture also tackles a complementary weakness. Convolutional neural networks excel at extracting local spatial detail, such as edges, textures and the shapes of individual objects, but their receptive fields limit their grasp of long-range relationships across a scene. Vision transformers, by contrast, use self-attention to model global context, letting every part of an image attend to every other part, but they can be less efficient at capturing fine-grained local structure. The proposed framework fuses convolutional blocks with Conv-ViT blocks, a hybrid in which convolutional operations and transformer attention are integrated so that local spatial details and global contextual dependencies are modeled jointly. This kind of hybridization reflects a broader trend in computer vision, following influential studies asking whether vision transformers see like convolutional networks and demonstrating that the two paradigms have complementary strengths.

Motion information enters the pipeline through a second clever design choice. Alongside the raw video frames, the researchers feed the network an additional input channel computed with Farneback optical flow, a dense optical flow algorithm that estimates the motion vector of every pixel between consecutive frames. Optical flow gives the model an explicit, pixel-level description of how everything in the scene is moving, independent of how it looks. By combining appearance information from the raw frames with motion information from the flow channel, the network can learn both appearance-based variations, such as an unexpected object, and motion-based variations, such as movement in the wrong direction or at an abnormal speed. This dual-stream strategy echoes earlier two-flow architectures but integrates the motion signal directly into a transformer-augmented reconstruction framework.

At inference time, the system scores each frame using the reconstruction error, complemented by the peak signal-to-noise ratio, a standard image quality metric that drops when a reconstruction deviates sharply from the original. Frames whose reconstructions are poor, or whose PSNR falls below the pattern established by normal footage, are flagged as anomalous. The evaluation was conducted on four widely used benchmarks: UCSD Ped1 and UCSD Ped2, which capture pedestrian walkways with cyclists, skaters and occasional vehicles intruding into the frame; the Avenue dataset, filmed in a campus entrance hall with loitering, throwing and running; and ShanghaiTech, a large and challenging collection of thirteen scenes with diverse camera angles and crowd conditions.

The results are striking. On UCSD Ped1 the framework achieved an area under the receiver operating characteristic curve of 98.97 percent with an equal error rate of 4.09 percent, meaning the point where its false alarm rate and miss rate cross sits below five percent. On UCSD Ped2 it reached 95.63 percent AUC with a 4.34 percent EER, and on Avenue it recorded 93.53 percent AUC with a 12.23 percent EER. On the hardest benchmark, ShanghaiTech, it attained 89.14 percent AUC with a 16.49 percent EER. The consistent performance across datasets with very different scene dynamics, lighting conditions and anomaly types suggests the hybrid design generalizes rather than overfitting to a single environment, and the reported equal error rates on the UCSD benchmarks place the method among the strongest reconstruction-based approaches described in the literature.

The implications extend well beyond academic benchmarks. Security operators, transit authorities and smart-city planners all face the same economics: footage is cheap, attention is expensive. Systems that can reliably narrow a human operator’s focus to the small fraction of video that actually deserves scrutiny could change how surveillance is staffed and reviewed. Because the framework is unsupervised, it also sidesteps the privacy and labeling burdens of supervised training, since it requires only examples of ordinary activity, which every camera already records in abundance. The authors note that the datasets used in the study are available from the first author on reasonable request, and the work was carried out without dedicated funding.

Challenges remain before such systems can be trusted in the wild. Real deployments must cope with camera shake, weather, gradual shifts in what counts as normal as seasons and crowds change, and the ethical questions that accompany any technology capable of deciding, autonomously, what counts as suspicious behavior. The equal error rates on the more crowded and heterogeneous benchmarks, while strong, still leave room for false alarms that could erode operator trust. Yet the trajectory is clear. By marrying the local precision of convolutions, the global reasoning of transformers, a memory that remembers how normal scenes unfold in time, and an explicit motion signal from optical flow, this work offers a blueprint for surveillance AI that watches quietly, learns the rhythm of a place, and speaks up only when the rhythm breaks.

Subject of Research: Unsupervised spatio-temporal video anomaly detection using a memory-augmented CNN-ViT autoencoder with Farneback optical flow

Article Title: Spatio-temporal video anomaly detection via CNN-ViT autoencoder and farneback optical flow

Article References: Pathak, V., Diwakar, M., Pandey, N. K., Roka, S., & Singh, P. (2026). Spatio-temporal video anomaly detection via CNN-ViT autoencoder and farneback optical flow. Cluster Computing, 29(13), Article 773. https://doi.org/10.1007/s10586-026-06598-5

Image Credits: AI Generated

DOI: 10.1007/s10586-026-06598-5

Keywords: video anomaly detection, surveillance, autoencoder, vision transformer, CNN, optical flow, Farneback, memory module, unsupervised learning, computer vision, deep learning, UCSD Ped2

The approach, called RMTA-Net, belongs to a family of techniques known as prediction-based unsupervised video anomaly detection. The core idea is elegant. The network is shown a short clip of video frames and asked to predict what the next frame will look like. Because it is trained only on normal scenes, it becomes an expert at forecasting ordinary motion and appearance. When an anomalous event unfolds, the model’s prediction diverges sharply from the actual frame, and the size of that discrepancy—measured pixel by pixel—becomes the anomaly score. Frames that deviate strongly from expectation are flagged for human review. This sidesteps the need for exhaustive annotation, since the model learns the statistics of normality from unlabeled footage, but it places enormous demands on the network’s ability to represent both what objects look like and how they move through time.

Existing prediction-based methods, the authors argue, fall short on two fronts. First, their spatial representations—the internal descriptions of shapes, textures, and objects in each frame—are often too coarse to capture subtle appearance variations. Second, they struggle to understand clip-level temporal dependency: the way motion in one frame conditions and constrains motion in the next. Complex scenes contain overlapping movements, occlusions, and camera perspective shifts that make naive temporal modeling brittle. A pedestrian partially hidden behind a van, a crowd flowing in two directions at once, or a vehicle entering the frame at high speed all require the model to reason jointly across space and time rather than treating each frame in isolation. RMTA-Net was designed specifically to close both gaps within a single architecture.

The first stage of the pipeline is a residual spatial feature enhancement network. Operating on individual frames, it progressively extracts structural and appearance information at multiple levels of abstraction, from fine edges and textures to coarse object layouts. The residual design means that each processing stage refines the features produced by the previous one, adding detail rather than replacing it, which helps preserve the subtle cues that distinguish a person from a shadow or a stroller from a bicycle. These independently encoded spatial features are then organized into an explicit temporal representation, effectively stacking the per-frame descriptions into a sequence that downstream modules can interrogate for motion patterns and inter-frame relationships.

The heart of the system is the module that gives the network its name: recurrent conditioned memory-guided temporal attention, or RMTA. This component fuses two ideas that have usually been pursued separately. The first is recurrent temporal processing, in which a recurrent network maintains a running internal state as it steps through the frames of a clip, carrying forward information about what has already happened. The second is a set of learnable memory banks—trainable storage that encodes dataset-level normality priors, essentially compressed prototypes of how normal scenes tend to look and move. During processing, the model performs attention-based memory access, querying the memory banks with the recurrent network’s output to retrieve relevant normality prototypes, while a parallel branch uses temporal attention to retrieve a global normality prior driven by the contextual relationships among frames in the clip.

Crucially, the two branches do not simply compete; they are combined through adaptive gating, a learned mechanism that decides, at each moment, how much weight to give the recurrent memory retrieval versus the contextual temporal relationships. When motion is regular and predictable, the recurrent pathway may dominate; when the scene is complex or the recurrent state is uncertain, the memory-guided global prior can compensate. This dual-branch design allows the model to maintain a robust sense of normality even under the appearance variations and tangled motion dynamics that degrade earlier methods. The fused representation is then passed to an attention-enhanced decoder, which reconstructs the future frame from the learned spatiotemporal embeddings. The decoder’s attention mechanism helps it focus on the regions of the scene most relevant to the prediction, sharpening the contrast between well-forecast normal content and the surprising content that signals an anomaly.

The authors evaluated RMTA-Net on three widely used benchmark datasets that together span the practical challenges of surveillance video. UCSD Ped2 contains pedestrian walkways filmed from a fixed viewpoint, with anomalies such as cyclists and small vehicles intruding into pedestrian-only spaces. CUHK Avenue features a campus avenue with more varied activities, including loitering, throwing objects, and unusual walking patterns. ShanghaiTech is the most demanding of the three, with thirteen scenes of differing complexity, crowded conditions, and diverse anomaly types. On these benchmarks, RMTA-Net achieved frame-level area-under-the-curve scores of 99.0 percent on UCSD Ped2, 90.1 percent on CUHK Avenue, and 75.8 percent on ShanghaiTech, remaining competitive with several state-of-the-art methods across the board. The AUC metric reflects how well the system ranks anomalous frames above normal ones, so scores approaching 99 percent on Ped2 indicate near-perfect discrimination in that relatively controlled setting, while the ShanghaiTech result demonstrates useful performance in genuinely cluttered, multi-scene environments.

What makes this result notable is not merely the headline numbers but the architectural lesson embedded in them. Memory-based reasoning gives the network something like an institutional memory of normality: rather than relying solely on the hidden state of a recurrent network, which must compress an entire clip into a fixed-size vector, the model can consult explicit prototypes of normal appearance and motion. Attention-guided retrieval means those prototypes are consulted selectively, only where the current context calls for them. The adaptive gate then arbitrates between memory and context on the fly. This division of labor mirrors, in a loose computational sense, the way long-term and episodic memory support expectations in human perception, a parallel the authors’ choice of terminology makes explicit. The result is a system whose predictions degrade gracefully under complexity rather than collapsing when scenes become crowded or visually ambiguous.

The practical implications extend well beyond benchmark scores. Because the method requires no anomaly labels, it can be deployed on the vast reservoirs of unlabeled footage that organizations already possess, learning the normal rhythm of a specific environment and flagging deviations without prior knowledge of what those deviations might be. That matters for applications ranging from public safety and traffic monitoring to industrial safety and elder care, where the set of possible incidents cannot be enumerated in advance. The open-access publication, with fees covered by the Government of India’s One Nation One Subscription initiative, means the full technical description is freely available to researchers and practitioners worldwide, lowering the barrier for replication and extension.

Challenges remain, as the authors’ own results make clear. Performance on ShanghaiTech, while competitive, still leaves substantial room for improvement in the hardest scenes, and future-frame prediction systems in general can be sensitive to camera jitter, illumination shifts, and the sheer diversity of real-world behavior. The field will also need to keep examining how such models behave across different populations and environments to ensure that flagged anomalies reflect genuine events rather than artifacts of the training distribution. Nevertheless, RMTA-Net offers a compelling demonstration that combining recurrent temporal modeling, learnable memory, and adaptive attention yields a more faithful model of normality—and that the better a machine understands the ordinary, the more sharply the extraordinary stands out. In surveillance, where the abnormal is rare by definition, that principle may prove to be the key to systems that finally earn the trust placed in the cameras they watch.

Subject of Research: Unsupervised video anomaly detection using future-frame prediction with memory-guided temporal attention

Article Title: RMTA-Net: recurrent conditioned memory-guided temporal attention with spatial feature enhancement for video anomaly detection

Article References: Chouhan, S. P., Punn, N. S., & Bhattacharya, M. (2026). RMTA-Net: recurrent conditioned memory-guided temporal attention with spatial feature enhancement for video anomaly detection. Complex & Intelligent Systems. https://doi.org/10.1007/s40747-026-02506-x

Image Credits: AI Generated

DOI: 10.1007/s40747-026-02506-x

Keywords: video anomaly detection, future-frame prediction, recurrent neural networks, memory banks, temporal attention, spatial feature enhancement, unsupervised learning, surveillance video, computer vision, deep learning, UCSD Ped2, ShanghaiTech