New Pixel-Level AI Model Gives Fine Control Over Image Generation

Anyone who has used an AI text-to-image generator such as ChatGPT or Google Gemini has likely run into a familiar frustration: the image looks nearly perfect, but asking the AI to fix one small detail causes it to mangle something unrelated. A team from the University of Rochester and NVIDIA says it has traced the problem to how these models are built, and has developed a new model that gives users control over individual pixels instead.

Most state-of-the-art AI image generators rely on diffusion models operating in latent space. In this approach, concepts such as a cat or a dog are compressed into their essential details and stored so they can be retrieved quickly and efficiently when it is time to create an image. That compression is part of what allows the models to generate the images a user describes within seconds, but it comes at a cost.

“Using the latent space helps cut down the computation, but the problem is that you lose controllability because you are not adjusting individual pixels,” said Jiebo Luo, the Albert Arendt Hopeman Professor of Engineering at the University of Rochester. “As a result, you can’t easily control specific parts of the images you generate.”

Luo’s lab and collaborators at NVIDIA were among the first teams to develop a model that trades the latent space for the pixel space. The model, called PixelDiT, was presented at the 2026 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), where it was nominated as a best paper finalist, an honor given to just 15 of more than 4,000 accepted papers at the conference. The model allows users to zero in on particular regions of an image, such as fixing only the pixels used to illustrate a pair of glasses on a person’s face or adjusting the color of a car.

The team has since built a more powerful successor, PixelDiT2, which implements pixel space diffusion in an efficient way. According to the researchers, PixelDiT2 achieves almost the same performance as state-of-the-art latent space models while providing controllability over individual pixels. A video released by the team shows the model generating images with near state-of-the-art quality while allowing edits at the level of single pixels. The team will present a paper showcasing PixelDiT2 at the Neural Information Processing Systems conference (NeurIPS) in December.

The work is a collaboration between Luo’s laboratory at Rochester and NVIDIA, the AI infrastructure company. The team includes computer science PhD student Yongsheng Yu, who worked on the project as an NVIDIA intern, and University of Rochester alumnus Wei Xiong ’22 PhD (computer science), now a senior researcher at NVIDIA.

With PixelDiT2 in hand, the researchers have turned their attention to an even more difficult problem: video generation. “Video is a big challenge because you have to worry about smooth transitions from frame to frame; otherwise it looks funny to the human eye,” Luo said. “You also have a lot more data to work with.”

But Luo said the progress the team has made with PixelDiT2 leaves them in a strong position to pursue AI video generation in the pixel space, which would provide users powerful new levels of control over the videos they generate.

The trade-off the researchers are working to resolve is fundamental to how modern generative image models are designed. Latent space diffusion made rapid, high-quality image generation practical by shrinking the amount of data the model must process, but that same compression blurs the link between the model’s internal representations and the specific pixels that make up the final image. Pixel-space approaches keep that link intact, allowing targeted edits, at the price of far heavier computation. By making pixel space diffusion efficient enough to nearly match latent space performance, PixelDiT2 suggests that fine-grained control and competitive image quality may not have to be mutually exclusive. A paper describing the work is available with the DOI 10.48550/arXiv.2609.24919.

Subject of Research: Pixel-space diffusion models for controllable AI image generation

Article Title: Improving AI-generated images, pixel by pixel

Article References: Improving AI-generated images, pixel by pixel. Original publication (Bibliographic details unavailable.)

Image Credits: AI Generated

DOI: Not provided

Keywords: AI image generation, diffusion models, pixel space, latent space, PixelDiT2, University of Rochester, NVIDIA, NeurIPS, CVPR, computer vision, controllability, video generation