Engineers have developed a scalable image compression architecture designed to simultaneously serve automated artificial intelligence models and human viewers without overloading network bandwidth. As connected infrastructure, municipal camera networks, automated vehicles, and personal monitoring devices proliferate across modern environments, massive volumes of visual data are routinely transmitted to backend servers. Most of these transmitted images are processed exclusively by deep learning networks executing computer vision tasks such as semantic segmentation and object classification, rather than being viewed by human observers. Standard compression schemes like JPEG, J2K, and BPG compress visual information according to fixed mathematical formulas or optimize exclusively for human perception metrics like Peak Signal-to-Noise Ratio. When traditional codecs or standard learning-based image compression frameworks optimize purely for visual appearance, they preserve unnecessary perceptual details while simultaneously discarding the compact, high-dimensional semantic features essential for accurate machine interpretation. To resolve this structural bottleneck, researchers Zhongpeng Zhang, Ying Liu, and Wen-Hsiao Peng introduced a framework termed Side Information-Driven Image Coding for Hybrid Machine–Human Vision, which decouples core visual recognition features from heavy pixel-level reconstructions.
The newly engineered architecture addresses foundational inefficiencies found across existing image coding for machines paradigms. Contemporary approaches typically compress images end-to-end, transmit them, decompress the files, and feed the reconstructed outputs into downstream analytical networks. Because these systems often optimize for human visual metrics rather than classification loss functions, the decoded outputs suffer from high spatial redundancy while losing structural cues required for high-precision machine recognition. Other specialized methods transmit standalone compressed latent representations directly to vision networks, but they operate as single-task systems that lack the flexibility to reconstruct visible images for human inspectors or support diverse recognition tasks. In contrast, the newly reported framework employs a scalable multi-layer hierarchy dividing transmission streams into a foundational base layer and a secondary enhancement layer. The base layer relies entirely on compact side information derived from hierarchical hyperprior tensors to execute image classification, semantic segmentation, and coarse image reconstruction. When high-fidelity human observation becomes necessary, an enhancement layer transmits a residual bitstream that refines the visual output into a sharp, high-fidelity image, preventing unnecessary data transmission across routine monitoring pipelines.
Within the base layer encoding stage, an input image passes through an initial neural encoder to produce a latent feature representation, which is subsequently processed by two sequential hyperprior encoders to generate two cascaded tiers of hyperprior side information. These three operational encoder modules each incorporate four convolutional layers. The primary feature extraction network employs generalized divisive normalization between convolutional layers, whereas the two subsequent hyperprior encoders utilize rectified linear unit activation functions. To efficiently transmit the shallower hyperprior features across bandwidth-constrained networks without sacrificing semantic precision, the pipeline incorporates a channel-wise auto-regressive hyper-prior context model. This context model partitions the quantized hyperprior tensor into discrete groups along the channel dimension. As entropy coding proceeds across each respective group, the system conditions probability distribution estimates on previously decoded channel groups alongside predictive parameters supplied by the deeper hyperprior layer, producing highly accurate mean and scale parameter predictions that minimize entropy coding overhead.
On the decoding side of the base layer, the researchers introduced a multi-scale feature fusion block specifically engineered to combine deep and shallow hyperprior streams into rich semantic representations for analytical models. Rather than relying on a single-stage adapter structure that merely adds upscaled deep features to shallow layers, the newly designed block implements three progressive fusion stages utilizing four-layer transposed convolutional structures. Deep hyperprior data is progressively upscaled through dedicated transposed convolutional decoders and iteratively summed with intermediate feature representations across multiple spatial scales. The intermediate layers leverage rectified linear units and inverse generalized divisive normalization to maintain feature stability across successive transformations. This progressive multi-scale combination enables analytical vision networks to directly interpret deep semantic boundaries without requiring intermediate spatial image decompression, maximizing rate-accuracy performance across downstream classification and segmentation benchmarks.
For complex scene understanding, the researchers conjoined their transmission framework with an upgraded semantic segmentation network designated as Modified TrSeg. Standard TrSeg architectures process input features through a deep convolutional backbone, pool multi-scale contextual features, and map source and target tokens through a transformer decoder before generating category prediction masks. Building upon the observation that binary class distinction is significantly more accurate than direct multi-class division, the authors modified this structure by repeating each channel of the initial semantic segmentation map 512 times to establish dedicated class blocks corresponding to specific visual categories. The multi-dimensional feature map is multiplied by these distinct class blocks to isolate class-specific sub-features, which then pass through point-wise convolutions to produce refined segmentation maps. When tested on the Cityscapes validation dataset at a standard resolution of 1024 by 2048 pixels, the modified segmentation architecture improved mean intersection over union by 2.14 percentage points over the baseline TrSeg model, confirming that class-isolated feature re-weighting provides superior structural delineation.
Extensive semantic segmentation evaluations conducted on the Cityscapes dataset demonstrated substantial transmission efficiencies compared to both conventional codecs and learned compression pipelines. When processing high-resolution 1024 by 2048 imagery, the new architecture achieved an equivalent mean intersection over union of approximately 51 percent using roughly 28 percent of the bitrate required by J-FT T-FT, approximately 20 percent of the bitrate of VVC-intra, and about 18 percent of the bitrate required by HEVC-intra. At an operational bitrate of approximately 0.11 bits per pixel, the proposed system attained a mean intersection over union exceeding 64 percent, outperforming VVC at 53.5 percent, HEVC at 51 percent, and both cheng20 and J-FT T-FT at roughly 48 percent. Furthermore, at an accuracy threshold of 61 percent mean intersection over union, the framework cut bitrate demands by more than half relative to JPEG AI. Similar coding gains were recorded on scaled 512 by 1024 inputs, where achieving an accuracy level of approximately 50 percent required roughly 0.02 bits per pixel, representing 45 percent and 60 percent bitrate savings against specialized benchmark codecs C128 and C64, respectively, while entirely eliminating the need to transmit external segmentation maps to decoders.
The framework exhibited comparable coding advantages when evaluated on large-scale image classification using the ImageNet1K benchmark, which comprises 1,000 object categories across 50,000 validation images. For classification tasks, the decoded multi-scale side information features were coupled directly into a Vision Transformer architecture operating with 24 by 24 patch windows on 384 by 384 inputs. By freezing the fundamental hyperprior encoding pipeline established during segmentation training and fine-tuning only the multi-scale fusion block and classifier parameters, the system achieved a top-one classification accuracy of approximately 75 percent at a transmission rate of 0.1 bits per pixel. Comparative evaluations against dedicated machine-oriented compression schemes, including SPIC-Q, Recognition-Aware learned compression, transformed image coding, and recurrent neural frameworks, confirmed that the proposed multi-scale hyperprior fusion mechanism consistently secured superior classification accuracy under lower data volume allocations.
Beyond machine vision execution, the base layer supports coarse image reconstruction exclusively from transmitted hyperpriors, producing preview imagery without activating residual networks. The system routes intermediate fused representations through specialized decoding blocks and a concatenation fusion module to reconstruct an initial visual estimate. On Cityscapes benchmarks scaled to 256 by 512 pixels, this coarse synthesis reached a 26-decibel Peak Signal-to-Noise Ratio at 0.08 bits per pixel, operating with 40 percent of the bitrate demanded by DSSLIC and standard JPEG, and just 25 percent of the bitrate required by RL-ASC. Because hyperprior data is fundamentally optimized for machine analytical discrimination rather than perceptual uniformity, standalone coarse reconstructions exhibit blurriness. To bridge this perceptual gap for human monitors, the enhancement layer encodes the spatial residual difference between the original input image and the coarse reconstruction using dedicated convolutional encoder and decoder pairings, transmitting quantized latent residuals to form fine reconstructed outputs.
When operating in fine reconstruction mode on Cityscapes imagery, the framework established superior human-oriented image fidelity across mid-range and higher bitrates. At an overall rate of 0.4 bits per pixel, the combined base and enhancement layers produced fine reconstructions exceeding 35 decibels in Peak Signal-to-Noise Ratio, surpassing both BPG and JPEG AI by more than 1 decibel. Perceptual side-by-side inspections at 0.3 bits per pixel confirmed that the reconstructed images exhibited sharper structural edges, superior color fidelity, and noticeably fewer blur artifacts than BPG baselines. However, experimental data revealed an inherent operational trade-off at very low bitrates ranging between 0.1 and 0.23 bits per pixel, where JPEG AI achieved superior reconstruction quality. Because the proposed architecture reserves a non-negotiable allocation of 0.08 bits per pixel for the base layer to prevent machine classification and segmentation degradation, the bit budget remaining for the residual enhancement layer becomes severely constrained under strict transmission caps.
The study also verified that modular component sharing substantially reduces overall computational complexity and storage overhead. In traditional deployments lacking architectural integration, supporting semantic segmentation, classification, coarse reconstruction, and fine reconstruction requires four independent models, accumulating a parameter and computational total of 4a plus 4b plus 2c plus d across individual functional blocks. By sharing frozen base encoders and intermediate decoders across tasks, the proposed architecture requires only a plus 3b plus c plus d in total resources, eliminating an overhead burden of 3a plus b plus c. Ablation studies systematically validating the multi-scale feature fusion block confirmed that every additional fusion step produced measurable increases in Cityscapes segmentation accuracy and ImageNet1K classification performance, establishing that multi-scale hyperprior integration provides an efficient, scalable foundation for hybrid machine-human visual infrastructures.
Subject of Research: Scalable image compression architecture for hybrid machine and human vision
Article Title: Side information-driven image coding for hybrid machine–human vision
Article References: Zhang, Z., Liu, Y., & Peng, W.-H. (2025). Side information-driven image coding for hybrid machine–human vision. EURASIP Journal on Image and Video Processing, 2025(1), Article 3. https://doi.org/10.1186/s13640-024-00661-0
Image Credits: AI Generated
DOI: 10.1186/s13640-024-00661-0
Keywords: image coding for machines, scalable coding, side information, hyperprior, semantic segmentation, image classification, Cityscapes, ImageNet1K, Modified TrSeg, multi-scale feature fusion, coarse image reconstruction, fine image reconstruction, deep learning compression, channel-wise auto-regressive model, rate-accuracy optimization, lossy compression, hybrid machine-human vision
Cite Scienmag News
Elena Sutton. (November 21, 2026). Dual-Layer Compression System Balances Machine Vision Tasks with Human Image Reconstruction. Scienmag. https://scienmag.com/dual-layer-compression-system-balances-machine-vision-tasks-with-human-image-reconstruction/
Elena Sutton. "Dual-Layer Compression System Balances Machine Vision Tasks with Human Image Reconstruction." Scienmag, 21 November 2026, https://scienmag.com/dual-layer-compression-system-balances-machine-vision-tasks-with-human-image-reconstruction/. Accessed 21 November 2026.
Elena Sutton. "Dual-Layer Compression System Balances Machine Vision Tasks with Human Image Reconstruction." Scienmag. November 21, 2026. https://scienmag.com/dual-layer-compression-system-balances-machine-vision-tasks-with-human-image-reconstruction/








