New AI Framework Lets Users Steer Image Generation With Text, Sketches and Feedback

Text-to-image systems have transformed how digital visuals are made, yet anyone who has wrestled with a diffusion model knows the frustration: a single text prompt rarely captures everything a creator wants, and when the first result misses the mark, the only option is often to start over from scratch. A new study published in Discover Artificial Intelligence tackles this problem head-on with a framework designed not just to generate images, but to hold a conversation with the person creating them. The work, led by Di Wu, Shuai Bai and Hailong Li of Shandong Huayu University of Technology, introduces a Multimodal Interactive Generation Framework, or MIGF, that combines text descriptions, reference images and structured inputs such as sketches into a single, iteratively refinable generation pipeline.

The core insight behind MIGF is that different sources of information are not equally reliable for every image. A sketch may pin down geometry precisely while saying almost nothing about color or texture; a reference photograph may be rich in visual detail yet contain elements irrelevant to the desired output. Most existing systems handle this by simply concatenating all the condition features or by applying a fixed set of learned weights that treat every input identically. MIGF’s Adaptive Condition Fusion module takes a different approach: it predicts, for each individual input, a normalized weight vector that determines how much each modality should contribute. The features are first contextualized through a self-attention operation, so the weight assigned to one condition can depend on what the other conditions already provide, and a lightweight gating network then produces the final modality weights that are injected into the denoising network through cross-attention.

This design has a practical consequence that the authors demonstrate directly. When a text prompt and a reference image already carry sufficient information, the framework can automatically down-weight a sparse or redundant sketch, rather than letting noisy structural cues corrupt the result. In a controlled comparison where only the fusion strategy was swapped while the backbone, training data and inference settings were held fixed, ACF outperformed naive concatenation, static weighted averaging and FiLM-style feature modulation on every metric measured, including image quality, semantic alignment and control precision. The authors are careful to note that the Softmax-based weighting bounds the fused feature norm, which helps explain the numerical stability of the mechanism, though they stop short of claiming it as a formal convergence guarantee.

The second pillar of the framework addresses a long-standing desire among digital artists: independent control over what an image shows versus how it looks. The Semantic Decoupling Module organizes the latent representation into two subspaces, one carrying content-related information such as structure and semantics, the other carrying style-related attributes such as appearance and tone. Training relies on contrastive supervision: images sharing the same content but rendered in different styles are treated as positive pairs in the content subspace, while semantically different samples serve as negatives, with an analogous objective for the style branch. Importantly, the authors frame this as practical separability rather than mathematically strict disentanglement, acknowledging that the two subspaces are encouraged to capture complementary factors without any hard independence constraint.

The third component, the Interactive Feedback Mechanism, is what turns the system from a one-shot generator into a genuine creative collaborator. Users can perform local modifications by selecting a region and providing a text instruction, adjust attributes such as style strength or color tone through a scalar control, or supply an additional reference image as guidance. Rather than treating each of these as a separate editing system, IFM converts all feedback into conditional signals compatible with the existing pipeline. A masked region is regenerated during subsequent denoising steps while the rest of the image is preserved, and the updated condition is blended into the original with a strength coefficient. This progressive adjustment strategy means successive user operations refine an existing result instead of restarting from an unrelated random sample, which is precisely the workflow designers and illustrators actually want.

To train and evaluate the framework, the team assembled a multimodal dataset of 50,000 high-resolution images spanning natural scenes, artistic works, design patterns and architecture, each accompanied by text descriptions, semantic annotations and structural information. Twelve trained annotators with design or computer vision backgrounds produced the labels under a unified protocol, with every sample reviewed by at least two annotators and disagreements adjudicated by senior reviewers. Inter-annotator agreement ranged from 0.79 to 0.86 across annotation types, indicating reasonably consistent labeling quality. Images were standardized at 512 by 512 pixels, with text descriptions averaging 24 words, and the data were split 8:1:1 into training, validation and test sets.

The headline numbers are striking. Under matched evaluation settings against ControlNet, the strongest self-run baseline, MIGF reduced the Fréchet Inception Distance, a measure of how closely generated images match the real distribution, by 23.7 percent, from 14.26 to 10.88. The CLIP Score, which quantifies text-image semantic consistency, improved by 18.6 percent, and the perceptual LPIPS metric dropped by 12.7 percent. Ablation experiments confirmed that each component contributes: adding ACF to the baseline cut FID from 16.42 to 13.75, adding SDM brought it to 12.19, and the full framework reached 10.88 with a CLIP Score of 35.7. When text, image and sketch conditions were combined, the framework achieved a Control Precision of 0.876, roughly an 11.9 percent improvement over the strongest single-modality configuration, sketch-only generation at 0.783.

The authors also took steps to guard against overclaiming. Comparisons were repeated across three random seeds with small standard deviations, and a zero-shot evaluation on 30,000 MS-COCO captions showed that the improvements transfer beyond the in-house data distribution, with MIGF achieving an FID-30K of 9.84. Results for closed-source systems such as DALL-E 2 and Imagen are reported only as contextual references, since identical inference conditions cannot be reproduced for them. Robustness testing revealed a nuanced picture: removing the reference image mainly hurt appearance quality, while degrading the sketch disproportionately damaged structural control, and simultaneous corruption of multiple conditions produced the largest performance drop, with FID rising to 12.71 and Control Precision falling to 0.792.

Speed matters for real creative work, and here the framework offers two operating points. The standard configuration uses 50 DDIM sampling steps and takes 4.3 seconds per image, while a fast mode requiring no retraining cuts the schedule to 20 steps and brings generation down to 1.9 seconds per image, roughly 0.53 images per second, with peak memory usage of 10.1 gigabytes. Quality curves show that most of the improvement occurs in the earlier sampling steps, so the fast mode trades a modest amount of fidelity for substantially lower latency. A sensitivity analysis of the five loss weights showed smooth performance around the selected configuration, suggesting the results are not an artifact of one fragile hyperparameter combination.

The authors are candid about limitations. Highly complex or internally contradictory prompts, such as classical futuristic architecture, can still produce semantic confusion, reflecting the difficulty of representing rare concept combinations with pretrained text-image representations. The interaction interface currently supports text, masks, sliders and reference images but not voice or gesture, computational cost remains non-trivial for edge deployment, and coverage of specialized domains like medical or satellite imaging is limited. Future directions include incorporating large language models to decompose complex instructions into structured constraints, extending the framework to video and 3D content, integrating watermarking for content provenance, and using the explicit per-modality weights as a starting point for interpretability analysis. Even with those caveats, MIGF offers a compelling demonstration that adaptive fusion, decoupled representations and unified feedback can be coordinated within a single diffusion pipeline, moving image generation closer to the iterative, multimodal way humans actually create.

Subject of Research: Multimodal interactive image generation using diffusion models with adaptive condition fusion, semantic decoupling and user feedback

Article Title: Generative models for interactive content creation and understanding in smart imaging

Article References: Wu, D., Bai, S., & Li, H. (2026). Generative models for interactive content creation and understanding in smart imaging. Discover Artificial Intelligence, 6(1), Article 1355. https://doi.org/10.1007/s44163-026-02418-2

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02418-2

Keywords: generative models, diffusion models, multimodal fusion, image generation, interactive AI, semantic decoupling, ControlNet, CLIP, content-style control, smart imaging, deep learning, human-AI interaction