The core idea of instruction-based image editing is deceptively simple. The system receives a source image and a natural-language instruction, and it must output an edited image that faithfully follows the instruction while leaving everything else untouched. The survey formally decomposes this task into two evaluation suites: a basic atomic suite covering fundamental, singular operations, and an advanced compositional suite targeting complex, multi-step reasoning. Together these suites span five major dimensions—object-level operations, image-level operations, utility tasks, complex instruction and reasoning, and spatial understanding and reasoning—encompassing twenty-one specific editing tasks. This hierarchical taxonomy fills a striking gap in a field where, until now, no unified definition of the task space existed, making it difficult to compare systems or diagnose their weaknesses systematically.

Architecturally, the field has passed through three distinct eras. Early systems, from roughly 2019 to 2023, leaned on Generative Adversarial Networks in combination with the CLIP vision-language embedding model. Techniques such as StyleCLIP and HairCLIP inverted real images into the latent space of a GAN and then manipulated latent vectors using directional CLIP similarity losses, allowing zero-shot edits of faces and hair. These methods were efficient and produced high-resolution outputs, but they were confined to narrow domains—faces, hairstyles, specific styles—and often suffered from optimization instability. The survey’s mathematical treatment makes the mechanics clear: a generator and discriminator play a minimax game, while text guidance enters through CLIP embedding differences that align image edits with textual intent.

The second era belongs to diffusion models, which have dominated since 2022. Latent diffusion architectures such as Stable Diffusion compress images into a lower-dimensional latent space via a variational autoencoder and then learn to reverse a gradual noising process, conditioned on text embeddings through cross-attention. The training objective is a denoising score matching loss that teaches the model to predict and remove Gaussian noise at every timestep. This framework spawned an enormous family of editing methods, which the survey divides into three classes: training-based approaches such as InstructPix2Pix and MagicBrush, which fine-tune diffusion backbones on large synthetic or human-annotated triplet datasets of source images, instructions, and edited targets; testing-time fine-tuning methods such as Imagic and Null-Text Inversion, which adapt a pre-trained model to each individual image for high-fidelity personalized edits; and training-free techniques such as Prompt-to-Prompt and DiffEdit, which intervene purely at inference time by manipulating attention maps or noise predictions to preserve unedited regions while applying the requested change.

The third and newest era, beginning around 2023, brings autoregressive models and unified multimodal frameworks. Autoregressive systems tokenize images and instructions into shared sequences and generate edits through sequential token prediction, factorizing the probability of each output token given all previous ones. Models such as CM3Leon, EditAR, and Nexus-Gen exploit this formulation’s natural strength in contextual reasoning and compositionality, though sequential decoding remains slower and prone to error accumulation. More ambitious still are hybrid unified systems such as Unified-IO 2 and Ovis-U1, which merge autoregressive and diffusion components to combine strong language understanding with high-quality image synthesis. The survey also examines commercial closed-source systems—GPT-4o, Gemini, Grok, SeedEdit 3.0, Qwen’s editing models, and Adobe Firefly—noting their end-to-end engineering advantages in multi-turn instruction following and built-in safety safeguards.

Perhaps the survey’s most illuminating contribution is its deep dive into training data construction. Because large-scale paired data of original images, instructions, and edited results barely exists naturally, researchers have engineered elaborate pipelines to synthesize it. For object removal, modular pipelines now use detection models like GroundingDINO and segmentation models like SAM to locate targets, followed by diffusion-based inpainting to fill the background. Addition tasks are frequently treated as the inverse of removal, with the edited image generated first and the original reconstructed afterward. Motion changes draw on video datasets such as Koala-36M, using optical flow estimation with RAFT and frame-pair descriptions from GPT-4o to create precise motion instructions. Attribute modification pipelines incorporate depth estimation for geometry-aware edits, while portrait enhancement still relies heavily on human curators because aesthetic standards are subjective and culturally variable. Every pipeline employs filtering—CLIP-based text-image similarity, L1 distance checks, and perceptual metrics—to discard misaligned or artifact-laden samples.

Evaluation has undergone its own quiet revolution. Traditional metrics split into two axes: instruction adherence, measured through CLIP directional and target similarities, and consistency preservation, measured through structural metrics like L1 distance and SSIM plus semantic metrics like LPIPS and DINO similarity. But the survey is blunt about their limits: these metrics trade off against each other in ways that penalize legitimate large edits, correlate poorly with human judgments, and cannot verify semantic subtleties such as object counts, spatial relations, or successful removals. This has driven a paradigm shift toward vision-language-model-based evaluation, exemplified by VIEScore and ImgEdit-Bench, which prompt multimodal models like GPT-4o to score edits on instruction adherence, edit quality, and detail preservation with natural-language rationales. Human evaluation, with its one-on-one comparisons and five-point rating scales, remains the gold standard against which all automated approaches are still calibrated.

The new CDD-IIE Bench aims to unify this fragmented evaluation landscape with 1,353 meticulously curated image-instruction pairs drawn from existing benchmarks and newly collected internet data, each pair verified by professional annotators. Twelve visual experts rated every model output across three dimensions per task, producing 162,360 individual expert judgments. The results are revealing. Qwen-Image-Edit-2509 and UniWorld-V2 topped the leaderboard with overall scores of 3.72 and 3.70, followed by Step1X-Edit-v1.2 and Flux.2-dev. But the most important finding is systematic: every model performs better on atomic edits than on compositional ones, and spatial understanding and reasoning is the weakest dimension overall, averaging just 2.92. Even the best model scored only 3.45 on spatial tasks. Models also show surprising weakness on utility tasks like deblurring and denoising, revealing a research bias toward high-level semantic editing at the expense of low-level fidelity that real-world applications demand.

These diagnostic results translate directly into a research agenda. The survey argues that closing the reasoning gap will likely require importing the reasoning paradigms of large language models into editing architectures, while robust generality across domains may call for modular designs that dynamically route tasks to specialized expert modules. Beyond reasoning, the authors identify several promising frontier directions: hybrid architectures in which diffusion models generate coarse layouts and autoregressive models refine details; latent-space autoregressive diffusion balancing efficiency and quality; distillation and sparse modeling for real-time editing on edge devices; multimodal inputs spanning sketches, audio, and 3D cues; and interactive refinement loops where users guide edits through conversational feedback. Equally pressing are ethical safeguards—watermarking and provenance tracking for AI-edited media, bias mitigation in training data, and adversarial robustness against malicious deepfake-style manipulation.

The pace of the field lends the survey particular urgency. The authors note that high-impact editing models and consumer applications have emerged almost monthly, with tools like ComfyUI and Stable Diffusion WebUI lowering technical barriers for creators while conversational assistants from OpenAI, Google, xAI, ByteDance, and Alibaba bake editing directly into chat interfaces. What was once the domain of professionals manipulating layers in Photoshop is becoming a universal language-based interaction. By imposing order on a rapidly expanding literature—over seventy rigorously instruction-following studies across GAN-CLIP hybrids, three diffusion paradigms, autoregressive systems, and unified multimodal giants—this survey provides both newcomers and specialists with a common map. Its central message is sobering and motivating in equal measure: the models can already perform remarkable single-sentence surgery on images, but teaching them to reason spatially, preserve fidelity, and generalize across every editing task remains the defining challenge of the next generation of research.

Subject of Research: A survey of instruction-based image editing covering data construction, model architectures, evaluation benchmarks, and applications

Article Title: Instruction-based image editing: a survey on data, models, evaluation, and applications

Article References: Zang, X., Jiang, Z., Cheng, J., Teng, Q., He, Y., Mu, Y., Ban, C., Zhang, H., Zhou, L., Feng, Z., & Zhang, C. (2026). Instruction-based image editing: a survey on data, models, evaluation, and applications. Vicinagearth, 3(1), Article 3. https://doi.org/10.1007/s44336-026-00034-3

Image Credits: AI Generated

DOI: 10.1007/s44336-026-00034-3

Keywords: instruction-based image editing, diffusion models, generative adversarial networks, autoregressive models, vision-language models, CDD-IIE Bench, Stable Diffusion, image inpainting, benchmark evaluation, multimodal AI, InstructPix2Pix, text-guided editing