A survey: advances in multi-modal visual understanding and generation

{
“title”: “How Diffusion Models Are Unifying Machine Vision, From Seeing Scenes to Imagining Them”,
“html”: “A comprehensive new survey published in the open-access journal Vicinagearth charts one of the most dramatic shifts in modern artificial intelligence: the rise of diffusion models as a single, unifying engine for both understanding and generating visual content. Authored by Dianbing Xi, Zhaoqi Zhou, Youming Wang, Jiepeng Wang, and Chi Zhang, with collaborators from the Institution of Artificial Intelligence (TeleAI) at China Telecom, Zhejiang University, and Shanghai Jiao Tong University, the review argues that the same mathematical machinery that powers photorealistic image generators is now being repurposed to estimate depth, predict surface geometry, segment objects, and even build navigable models of the physical world. The convergence, the authors contend, points toward a future in which machines perceive and imagine through one shared framework, much as humans do.

The technical heart of this transformation lies in how diffusion models work. Unlike earlier generative systems that produced images in a single step, diffusion models learn to reverse a gradual noising process, starting from pure randomness and iteratively denoising their way toward coherent structure. Denoising Diffusion Probabilistic Models, introduced in 2020, demonstrated that this stochastic procedure could outperform generative adversarial networks on image synthesis, and a wave of refinements followed. Most modern systems operate not on raw pixels but in a compressed latent space produced by a variational autoencoder, a strategy popularized by Latent Diffusion Models and their descendants Stable Diffusion and SDXL. This compression dramatically reduces computational cost while preserving fidelity, making high-resolution generation practical on consumer hardware.

Architecturally, the field has since moved beyond the convolutional U-Net backbone that defined the first generation of diffusion systems. The Diffusion Transformer, or DiT, replaces the U-Net with a scalable transformer and underpins flagship text-to-image systems such as PIXART-alpha, PIXART-Sigma, and the 4K-capable Wuerstchen pipeline. In parallel, flow matching techniques inspired by Rectified Flow learn a more direct trajectory from noise to image, enabling remarkably fast sampling. InstaFlow showed that a single denoising step could suffice for high-quality text-to-image synthesis, and the state-of-the-art SD3 model combines both trends by pairing rectified flow with transformer backbones. The latest generation, exemplified by FLUX.1 Kontext, unifies image generation and editing within a single flow matching architecture, maintaining multi-turn consistency as users iteratively refine their creations.

Video generation has followed a parallel but more demanding trajectory, because adding a temporal dimension multiplies computational complexity and introduces the problem of temporal coherence. Early systems such as Video Diffusion Models extended 2D U-Nets to 3D by inserting temporal attention layers, while Make-A-Video, Imagen Video, and MagicVideo demonstrated that large-scale image priors could be transferred to video with relatively little paired text-video data. Latent-space approaches like LVDM compressed video into low-dimensional representations for efficient long-form generation, and Stable Video Diffusion scaled latent video diffusion to large datasets by building on pretrained image models. More recently, transformer-based architectures including CogVideoX, LATTE, and OpenAI’s Sora have adopted the Diffusion Transformer to better fuse linguistic semantics with long-range motion, while industrial-scale systems such as HunyuanVideo, Kling, Mochi, Wan, and SkyReels-V1 continue to push quality and generalization boundaries.

Yet text prompts alone proved too blunt an instrument for creators who need precise control over spatial layout, human pose, or camera motion. The survey highlights how researchers answered with structured conditioning. ControlNet, perhaps the most influential contribution, attaches lightweight zero-initialized branches to a frozen diffusion model, translating depth maps, edges, poses, and segmentation masks into feature-space guidance without disturbing the base model’s knowledge. Video systems like Gen-1 decoupled structure from content, using monocular depth as a motion scaffold and CLIP embeddings as appearance signals. A second wave of 3D-aware methods goes further, injecting camera trajectories as ray embeddings or generating coarse point clouds as geometric scaffolds before refinement. Systems such as CameraCtrl, ViewCrafter, GEN3C, and Uni3C enable explicit viewpoint control and novel view synthesis, opening applications in filmmaking, simulation, and robotics.

What makes the survey striking is its documentation of the reverse direction: diffusion models being turned into perception engines. Rather than classifying or regressing, generative approaches now treat understanding tasks as conditional generation problems. Marigold reframes monocular depth estimation as a latent diffusion task and achieves strong zero-shot performance by repurposing pretrained image generators. DepthFM accelerates the idea with flow matching, while StableNormal and NormalCrafter generate temporally consistent surface normal maps from images and video. Segmentation has followed suit: DiffusionInst and ODISE formulate instance and panoptic segmentation as conditional generation, refining masks through denoising, and training-free methods show that powerful semantic cues are already embedded in pretrained diffusion backbones, extractable through feature clustering and attention analysis without any task-specific training. Joint frameworks like GeoWizard, Lotus, GeometryCrafter, Geo4D, and DICEPTION extend this to simultaneous prediction of depth, normals, camera parameters, and point clouds, often surpassing discriminative specialists in zero-shot and cross-domain settings, though the multi-step denoising process still imposes a significant inference cost.

The survey also maps how models are learning to generate multiple modalities at once, moving beyond RGB toward outputs that carry explicit geometry. Two architectural strategies dominate. Multi-modal VAE approaches, exemplified by Orchid and Trellis, encode RGB, depth, and normals, or point clouds and semantic features, into a shared latent space decoded into different modality-specific outputs, enabling coherent joint synthesis for 3D asset creation and sensor fusion. Shared VAE approaches instead pass all modalities through a single autoencoder: Matrix3D performs pose estimation, depth prediction, and novel view synthesis in one masked multi-modal diffusion transformer; VideoJAM jointly models appearance and optical flow with an inner-guidance mechanism for temporally coherent motion; Voyager synthesizes aligned RGB and depth video for unbounded, world-consistent 3D scene expansion; and JointDiT captures the joint RGB-depth distribution through adaptive weighting. Together they signal a shift toward flexible latent frameworks that can mask, fuse, and weigh heterogeneous signals spanning appearance, geometry, semantics, and motion.

At the frontier lies the boldest ambition: unified models that generate and understand in a single system. The authors trace a lineage from masked-autoencoder and tokenized approaches such as MultiMAE, UNIFIED-IO and its sequel, the 4M family, Sapiens, and PixelWorld, which consolidate dozens of tasks and modalities into shared token spaces. Diffusion-based successors now close the loop. UniReal treats image tasks as discontinuous video frames, OneDiff frames tasks as views at different noise levels, MMGen and OmniGen2 unify generation, editing, and understanding with parallel transformers, and on the video side OmniVDiff jointly models RGB, depth, segmentation, and edges through a shared 3D VAE, while Aether demonstrates zero-shot 4D reconstruction and goal-driven planning, and VACE consolidates creation and editing operations from move-anything to animate-anything in one latent diffusion framework. Applications already span world models for autonomous driving such as STAG-1 and Cosmos-Transfer1, immersive 3D and 4D scene generation via Matrix3D and Geo4D, and flexible video editing pipelines.

Challenges remain substantial, the survey cautions. How modalities interact during joint training is poorly understood, with some signals helping and others injecting noise or redundancy. Unified models still trail unimodal specialists on per-modality accuracy, slow multi-step inference limits deployment, and long-range video with temporally consistent multi-modal coherence remains unsolved. Bridging synthetic generation with real-world interaction, essential for embodied AI and robotics, is the ultimate test. Still, the trajectory is unmistakable: diffusion models have evolved from novelty image generators into a general substrate for visual intelligence, and the survey’s vision of systems that simultaneously perceive, reason about, and imagine the world is no longer science fiction but an active engineering frontier, one whose benchmarks, efficiency innovations, and unified architectures will shape the next decade of computer vision research.

“,
“excerpt”: “A new survey in Vicinagearth maps how diffusion models are converging into unified frameworks that both generate and understand images, video, and 3D geometry.”,
“subject”: “Diffusion-based multi-modal visual understanding and generation”,
“tags”: [“diffusion models”, “text-to-image generation”, “text-to-video generation”, “controllable generation”, “depth estimation”, “surface normal prediction”, “segmentation”, “multi-modal generation”, “diffusion transformer”, “unified vision models”, “3D scene generation”, “world models”]
}
“Subject of Research: Diffusion-based multi-modal visual understanding and generation

Article Title: A survey: advances in multi-modal visual understanding and generation

Article References: Xi, D., Zhou, Z., Wang, Y., Wang, J., & Zhang, C. A survey: advances in multi-modal visual understanding and generation. Vicinagearth, 3, 8 (2026). https://link.springer.com/article/10.1007/s44336-025-00032-x

Image Credits: AI Generated

DOI: 10.1007/s44336-025-00032-x

Keywords: diffusion models, text-to-image generation, text-to-video generation, controllable generation, multi-modal generation, depth estimation, segmentation, diffusion transformer, unified model, 3D scene generation, world models, visual understanding

Subject of Research: A survey: advances in multi-modal visual understanding and generation

Article Title: A survey: advances in multi-modal visual understanding and generation

Article References: Xi, D., Zhou, Z., Wang, Y., Wang, J., & Zhang, C. (2026). A survey: advances in multi-modal visual understanding and generation. Vicinagearth, 3(1), Article 8. https://doi.org/10.1007/s44336-025-00032-x

Image Credits: AI Generated

DOI: 10.1007/s44336-025-00032-x

Keywords: survey, advances, multi-modal, visual, understanding, generation, scientific research