Curriculum-trained GAN teaches steganography to hide data with pentary pixel changes

Researchers in China have developed a new artificial intelligence framework that teaches steganographic systems how to hide secret messages inside images more securely than ever before, particularly when the hidden payload is large. The work, published in the journal Cybersecurity, tackles a long-standing limitation of machine-learning-based steganography: most existing generative adversarial networks can only learn to modify pixel values by at most one unit, a constraint that becomes increasingly costly as the amount of hidden data grows.

Steganography differs fundamentally from encryption. Encryption scrambles the content of a message so that intercepted text is unreadable, but the very existence of the secret communication remains visible to an observer. Steganography goes one step further and conceals the existence of the communication itself. Digital images are the most widely used carriers for this covert channel because they are ubiquitous and highly redundant: tiny changes to pixel values, invisible to the human eye, can carry hidden information while the resulting stego image remains visually indistinguishable from the original.

Modern image steganography predominantly follows the distortion-minimization framework, in which every possible embedding unit is assigned a modification cost. Coding schemes such as syndrome-trellis codes, steganographic polar codes, and low-density generator-matrix codes then use these costs to achieve near-optimal embedding under payload constraints. The design of the embedding costs directly determines where and how pixels are modified, and it is therefore central to steganographic security. Handcrafted cost functions such as HUGO, WOW, HILL, and S-UNIWARD rely on heuristic priors to concentrate modifications in textured regions, but their fixed assumptions limit adaptability to modern convolutional neural network steganalyzers such as Xu-Net, Yedroudj-Net, SRNet, Ke-Net, and CovNet, which have dramatically improved detection performance.

This performance ceiling has motivated data-driven approaches in which embedding costs are learned end-to-end through adversarial optimization. Methods such as ASDL-GAN, UT-GAN, SPAR-RL, and Steg-GAN have demonstrated that generative adversarial networks can learn distortion functions automatically, adapting to whatever steganalyzer they face. However, nearly all of these GAN-based cost learners are confined to ternary embedding, meaning pixels can be left unchanged or shifted by plus or minus one. Earlier theoretical work by Sedighi and colleagues showed that allowing pentary modifications, in which pixels can also shift by plus or minus two, improves empirical security over ternary embedding, especially at higher payloads. Their approach, however, derived optimal pentary costs from an explicit statistical model of local image neighborhoods, which still relies on fixed assumptions.

Extending GAN-based cost learning to pentary embedding is difficult for three reasons. First, the search space expands from three possible states per pixel to five, making optimization much harder and slowing convergence. Second, the interactions between different modification levels make the joint allocation of small and large changes difficult to optimize. Third, the resulting optimization landscape is too complex for networks to converge reliably when all modification levels are trained simultaneously. The new framework, called Curriculum-based Pentary Embedding GAN, or CuPE-GAN, addresses these challenges by decomposing pentary embedding into two independent ternary surrogate layers. The first layer controls the primary plus-or-minus-one modifications, while the second layer introduces additional plus-or-minus-one changes whose superposition enables plus-or-minus-two modifications. This decomposition converts a direct five-state optimization problem into two progressively optimized ternary subproblems.

The training strategy is inspired by curriculum learning, the principle that models learn better when tasks are introduced in order of increasing difficulty. During the first stage of training, the second-layer channel is disabled and the model learns only the fundamental cost patterns of ternary embedding, concentrating modifications in textured regions. Once the first-layer allocation has settled, the second layer is activated and the generator learning rate is forcibly decayed to eighty percent of its previous value to prevent the new gradients from destroying the representations learned in stage one. The stage transition was set at seventy-two epochs using a scheduler-aligned heuristic: the first stage must pass the first three learning-rate decay points at epochs twenty, forty, and sixty, after which twelve additional epochs at the reduced learning rate are retained before the higher-order channel is introduced. This leaves seventy-eight epochs for pentary exploration and keeps the optimization budgets of the two stages approximately balanced.

A second technical innovation is the Improved Decoupled Ternary Approximation, or IDTA, a differentiable embedding simulator that represents the positive and negative modification directions using two explicit one-sided soft indicators built from hyperbolic tangent functions. The key property of IDTA is exact zero activation at finite sharpness: when the probability of one modification direction is zero, the corresponding soft branch is exactly inactive, whereas the widely used Double-Tanh simulator retains a small boundary residual that decays only as the sharpness parameter grows. Because the residual behaves like a constant divided by the sharpness parameter, it can perturb the training signal near sampling boundaries. IDTA does not change the limiting sampling law as sharpness approaches infinity; instead, it provides a branchwise differentiable approximation with improved finite-sharpness boundary consistency, which the authors found beneficial for pentary cost learning in controlled comparisons.

During adversarial training, the generator uses a modified U-Net architecture with eight downsampling modules and a decoder with two parallel branches that output the dual-layer probability maps through sigmoid activation. The discriminator combines two steganalyzers, one based on Xu-Net and one on Yedroudj-Net, with an adaptive update strategy that refreshes only the weaker steganalyzer at each iteration while the generator optimizes against the stronger one. A direction consistency loss penalizes cases where the two layers would modify a pixel in opposite directions, which would cancel out and waste embedding capacity while introducing unnecessary statistical anomalies. A sparsity penalty keeps second-layer modifications rare and concentrated in highly complex texture regions. After training, the learned dual-layer probabilities are synthesized into a normalized five-symbol distribution and converted into pentary embedding costs through a Gibbs distribution, and the actual message embedding is performed by a standard pentary syndrome-trellis code encoder rather than by directly superimposing two ternary modifications.

The model was trained once on a mixed dataset of forty thousand grayscale images of size 256 by 256, combining thirty thousand high-quality facial images from CelebA-HQ with ten thousand diverse natural scene images from ALASKA1 to enhance generalization. Evaluation was then performed directly on two standard steganalysis benchmarks, BOSSBase and BOWS2, each containing ten thousand grayscale images of size 512 by 512 with different statistical characteristics. Security was quantified by the detection error rate, the average of the false alarm and missed detection rates, where fifty percent indicates that the steganalyzer degenerates to random guessing. Five steganalyzers were used for evaluation: the traditional SRM rich model and four deep learning methods, Ke-Net, Yedroudj-Net, EfficientNet-B2, and CovNet. Uncertainty from the finite test set was quantified with a paired non-parametric bootstrap that resampled the held-out cover-stego pairs ten thousand times while preserving correspondence.

The results show that CuPE-GAN achieves higher average detection error rates than Steg-GMAN, WOW, and S-UNIWARD at medium and high payloads. At low payloads of 0.2 bits per pixel, performance differences among methods are small because limited modifications allow all methods to effectively conceal their traces. But as the payload increases, the advantages of pentary embedding become apparent. At 0.4 bits per pixel, CuPE-GAN reaches an average detection error rate of 41.42 percent on BOSSBase, improving by 6.16 percentage points over Steg-GMAN and 8.33 percentage points over S-UNIWARD. At 0.8 bits per pixel, the gains relative to Steg-GMAN are 5.23 percentage points on BOSSBase and 7.92 percentage points on BOWS2. Generalization was particularly striking: traditional methods suffer severe degradation on BOWS2 at high payloads, with S-UNIWARD dropping to an average detection error rate of only 6.54 percent at 0.8 bits per pixel, while CuPE-GAN maintains 17.53 percent. Ablation experiments confirmed that the curriculum-based dual-layer strategy outperforms both direct pentary cost learning and dual-layer decomposition without curriculum learning, validating the progressive training design. The authors plan to extend the framework to the JPEG domain and to incorporate attention mechanisms into the generator to better capture the spatial correlation of neighboring pixel modifications, opening the door to broader practical deployment of high-capacity covert communication.

Subject of Research: Learning pentary embedding costs for enhancing image steganography security via curriculum-based GAN

Article Title: Learning pentary embedding costs for enhancing image steganography security via curriculum-based GAN

Article References: Wang, X., Fu, H., Cao, Y., & Zhu, M. (2026). Learning pentary embedding costs for enhancing image steganography security via curriculum-based GAN. Cybersecurity, 9(1), Article 243. https://doi.org/10.1186/s42400-026-00669-1

Image Credits: AI Generated

DOI: 10.1186/s42400-026-00669-1

Keywords: steganography, pentary embedding, generative adversarial networks, curriculum learning, image security, steganalysis, Learning, pentary, embedding, costs, enhancing, image