Patch Shuffling Trick Makes Adversarial Attacks on Vision Transformers Far More Transferable

Adversarial examples that fool one artificial intelligence model often fail against another, but a new attack method shows that a surprisingly simple manipulation inside vision transformers can dramatically change that. In a study published in Discover Computing, researchers introduced Patches Restructuring, or PR, a plug-in strategy that randomly shuffles the order of embedded patch tokens during adversarial example generation on vision transformer surrogate models. The approach leaves the input image, model parameters, and patch embedding layer untouched, yet it substantially improves how often adversarial examples crafted on one model fool entirely different architectures, including convolutional networks, multilayer perceptrons, and adversarially trained defense models.

Vision transformers process images very differently from the convolutional neural networks that dominated computer vision for years. Instead of scanning local neighborhoods with convolutional kernels, a vision transformer splits an image into a sequence of patches, embeds each patch as a token, and uses multi-head self-attention to model relationships among all patches at once. This global receptive field is one reason vision transformers achieve strong recognition performance, but it also changes how they behave under adversarial perturbation. According to the authors, during adversarial optimization the self-attention mechanism can repeatedly emphasize patches that are not aligned with the actual object in the image, so the optimized perturbation drifts toward local or less informative patches. The result is a form of local patch overfitting that weakens transferability, meaning adversarial examples crafted on the surrogate model fool fewer unseen target models.

Transferability is the backbone of practical black-box attacks. In a white-box setting, an attacker can read model internals such as parameters and gradients, but in a black-box scenario only limited information like model outputs is available, which is closer to real deployment conditions. Transfer-based attacks solve this by crafting adversarial examples on a locally available surrogate model and hoping the perturbations also mislead unseen target models. Many influential transfer attacks, including the momentum iterative method MIM and the diverse input method DIM, were originally designed for convolutional networks, relying on gradient stabilization, input diversity, or other data augmentation in pixel space. The study argues that directly transplanting these CNN-oriented strategies to vision transformers is not necessarily effective, because the patch-based input representation, global receptive field, and attention-driven feature interaction of transformers produce different adversarial behaviors.

The new method works at a level that earlier attacks did not touch. For each forward pass and each attention layer during adversarial example generation, PR is activated with a configurable probability. When activated, the embedded patch tokens are randomly shuffled while the class token stays in place, and the restructured sequence is fed into the standard attention computation that produces the query, key, and value matrices. If the probability draw does not trigger the shuffle, the original token order is used. Crucially, the semantic representation of each patch token remains intact; only the order in which tokens participate in attention changes. The authors emphasize that PR does not modify positional embedding parameters, layer normalization, the multilayer perceptron block, or residual connections, and it keeps the standard attention computation unchanged, introducing stochasticity only before the query, key, and value projections.

The mechanism behind the improvement is diversity in the attention layer itself. Because the query, key, and value matrices are computed from the embedded patch sequence, their values change when the sequence order changes, which in turn diversifies the attention map and the feature vectors output by the attention layer. During iterative adversarial optimization, the attacker no longer relies on a single fixed patch order in every forward pass; gradients are instead computed under different randomly shuffled patch arrangements. This reduces the dependence of the generated perturbation on fixed local patch relations and helps alleviate local patch overfitting, which the authors identify as the key to stronger black-box transferability. Feature map visualizations on a T2T-ViT-24 model show that random restructuring disrupts the original patch distribution, yet the self-attention mechanism still captures relationships between critical patches, prompting attention to focus on crucial regions.

The empirical case for PR rests on a broad evaluation across 20 classification models built from a 1,000-image subset of the ImageNet validation set, using images that all evaluated models classify correctly. The target pool included 8 vision transformers, 4 convolutional networks, 4 multilayer perceptron architectures, and 4 adversarially trained CNN defense models. All experiments used a maximum perturbation of 16, 50 iterations, and a step size of 16/25, with attack success rate as the primary metric for untargeted black-box transferability. The numbers are striking in specific cross-architecture cases. When attacking from a T2T-ViT-24 surrogate toward ViT-B/16, the baselines MIM, DIM, and their combination MITIDIM transferred only 6.60 percent, 8.70 percent, and 25.40 percent of adversarial examples respectively, while combining MITIDIM with PR reached 42.50 percent. Against ResNet50, the same three baselines achieved 20.10 percent, 25.40 percent, and 45.10 percent, but MITIDIM plus PR jumped to 79.50 percent. Against MixerMLP-L/16, the baselines managed 10.50 percent, 13.30 percent, and 23.70 percent, while the PR combination reached 52.20 percent.

The method also held its own against attacks designed specifically for transformers. Using ViT-B/16 as the source model, the researchers compared PR with Self-Ensemble, which builds an ensemble from class tokens at different layers with a shared classification head, and Pay No Attention, which skips attention gradients during backpropagation. When combined with MIM, PR achieved higher transferability than both MIM plus Self-Ensemble and MIM plus Pay No Attention across target models spanning a CNN, a vision transformer, an MLP architecture, and a defense model. PR could additionally be combined with Self-Ensemble and Pay No Attention themselves for further gains. The authors are candid that the improvement is not universal: PR does not monotonically improve every source-target pair, and in a small number of cases, such as a performance decrease on Swin-L when PR was combined with DIM and Pay No Attention, additional patch shuffling occasionally interfered with an already strongly modified gradient direction. On average, however, PR was beneficial across the evaluated black-box settings.

Defense models proved no safe harbor. Against four adversarially trained models, Adv-EfficientNet-b0, ResNet152_denoise, ResNet50_FreeAT, and ResNet50_FastAT, using an ensemble of four vision transformer source models, PR showed strong attack performance, indicating that adversarial examples generated with the method retain high transferability even against models trained specifically to resist such perturbations. Visual analysis reinforced the qualitative story. Grad-CAM images captured every five rounds during 50-iteration attacks on a T2T-ViT-24 source model showed that with plain MIM, the region of interest in the source model gradually drifted away from the object itself, failing to concentrate on the areas the ResNet50 target model found important. Self-Ensemble kept attention on the object, but the PR combination covered a broader area of the object and destroyed object features more thoroughly. Perturbation visualizations on DeiT-B/16 similarly suggested that PR produces perturbations more concentrated around visually discriminative regions than MIM or DIM, and multi-scale structural similarity scores indicated that PR-generated adversarial examples were of higher perceptual quality than baseline outputs.

A sensitivity analysis of the shuffling probability revealed an important constraint. Testing with MIM across 11 vision transformer source models and reporting the average attack success rate on the 20 target models, the researchers found that transferability first increased and then decreased as the probability grew, with nearly all architectures showing the same trend. Values between 0.15 and 0.5 were most beneficial, while an excessively large probability did not contribute to high transferability and instead significantly reduced model accuracy, rendering attacks impractical. Different vision transformers showed different sensitivities, so the optimal probability varies by model. The authors also note that the method is lightweight and complementary rather than a replacement for existing approaches: unlike input-transformation attacks such as DIM and TIM, which resize, pad, or translate pixels, PR operates on the embedded patch sequence inside the surrogate, and unlike token-refinement methods, it simply shuffles token order with a probability.

The study’s limitations frame how far these results extend. The evaluation focused on digital-domain ImageNet classification, and extending PR to tasks such as object detection, fine-grained recognition, or physical-world attacks would require further evaluation under changes in scale, viewpoint, illumination, printing noise, and imaging distortion. Because PR involves stochastic patch shuffling, the authors note that repeated runs with different random seeds would be useful for assessing stability, and they leave a more systematic statistical evaluation to future work. Even so, the findings mark a notable shift in perspective for adversarial attack research: rather than treating vision transformers as awkward targets for CNN-era tricks, the work demonstrates that the internal patch structure of transformers is itself a productive surface for attack design, and that a lightweight, probability-triggered shuffle inside the surrogate forward pass can turn one of the field’s most persistent weaknesses, poor cross-architecture transferability, into a substantially smaller problem.

Subject of Research: A lightweight patch-shuffling strategy that improves the transferability of adversarial examples generated on vision transformer surrogate models against diverse target architectures and defenses.

Article Title: Transferable adversarial attacks on vision transformers via Patches Restructuring

Article References: Zhao, Y., Zhou, H., Fan, Q., Xie, H., Pan, Z., Wang, Y., Tan, Y.-A., Ying, Z., & Zhou, W. (2026). Transferable adversarial attacks on vision transformers via Patches Restructuring. Discover Computing, 29(1), Article 824. https://doi.org/10.1007/s10791-026-10735-4

Image Credits: AI Generated

DOI: 10.1007/s10791-026-10735-4

Keywords: adversarial examples, vision transformers, transferability, black-box attacks, self-attention, patch shuffling, ImageNet, machine learning security, adversarial robustness, surrogate models, cross-architecture attacks, defense models