AI Learns to Steal the Camera Work of Any Movie Clip and Apply It to Your Photos

Anyone who has tried to direct an AI video generator knows the frustration. You type “zoom in slowly” or “pan across the scene,” and the model delivers something close, but never quite the sweeping dolly shot or the tense, creeping push-in you saw in a favorite film. Camera motion, the invisible hand that guides perspective and emotion in cinema, has remained stubbornly out of reach for casual creators. Now a team of researchers from the University of Maryland and Dolby Laboratories has built a system that lets anyone borrow the exact camera movement from a single reference video and apply it to their own still image, with no 3D reconstruction, no trajectory files, and no technical expertise required.

The work, published in Discover Artificial Intelligence, addresses what the researchers call the “expressive gap” between a creator’s cinematic vision and the blunt instruments available in generative tools. Text prompts are too abstract to capture nuanced motion, while professional motion panels demand calibration skills and patience that most people simply do not have. The new framework, described as zero-shot personalized camera motion control, works by example: users supply a short clip whose camera behavior they admire, along with their own static image, and the system generates a video in which the user’s scene moves exactly the way the reference clip did.

Technically, the method is a two-phase pipeline built on top of a pretrained text-to-video diffusion model. In the first phase, the system performs inference-time optimization using two sets of Low-Rank Adaptation networks, or LoRAs, inserted into the model’s UNet architecture. Spatial LoRAs, placed in the spatial self-attention layers, learn the visual appearance of scenes, while temporal LoRAs, placed in the temporal self-attention layers, capture the camera motion itself. Cross-attention layers that bind text tokens to visual features remain frozen throughout. The training proceeds in two stages: first, the temporal LoRAs learn the motion dynamics of the full reference video while the spatial LoRAs learn generic appearance from randomly sampled single frames, discouraging overfitting to any one moment; second, the temporal LoRAs are frozen and the spatial LoRAs are re-tuned to represent the user’s target image.

The crucial innovation is an orthogonality regularizer that prevents these two sets of learned features from interfering with each other. When the spatial LoRAs are adapted to a new image, they risk overwriting shared low-level representations in a way that suppresses the frozen temporal LoRA’s motion signal at inference time. To prevent this, the researchers introduced a loss function that penalizes alignment between the spatial and temporal LoRA weight directions, comparing only the most significant singular directions obtained through truncated singular value decomposition. Minimizing this inner product encourages the two updates to occupy near-orthogonal subspaces, so that adapting the appearance of one scene does not erode the camera motion learned from another.

The second phase tackles a subtler problem: even a well-tuned diffusion model is not explicitly constrained by geometry, so the generated video can drift from the intended camera path. The researchers therefore added homography-guided inference, borrowing a classical computer vision technique that computes the planar transformation mapping one image onto another. Using SIFT keypoint matching and RANSAC, the system extracts frame-to-frame homography matrices from the reference video, warps the user’s image into a weak geometric estimate of the desired sequence, and then nudges the diffusion model’s predicted latents toward that estimate at every denoising step. This weak guidance keeps the output anchored to both the user’s scene and the reference motion without requiring any 3D reconstruction or camera pose estimation.

The choice of homographies over more elaborate 3D methods is deliberate. Tools like COLMAP, which recover camera trajectories from structure-from-motion, fail when a zoom effect is achieved purely by changing focal length rather than physically moving the camera, and they struggle with flat or textureless scenes. Homographies, by contrast, can be computed from almost any footage and capture the relative planar transformations, panning, tilting, and zooming, that define most cinematic camera work. The same insight underpins the team’s second contribution: a new evaluation metric called CameraScore, which measures the squared difference between homography matrices computed from consecutive frames of the reference and generated videos, providing a computationally cheap and scene-independent way to judge motion fidelity across structurally different scenes.

To test the system, the researchers curated a dataset of 680 reference video and user scene combinations, spanning pans, zooms, tilts, dolly shots, and 3D rotations drawn from movies, documentaries, and animation, paired with target scenes ranging from landscapes to urban settings. Quantitative comparisons against zero-shot baselines, including a naive homography approach, a DreamBooth-modified Tune-A-Video, and MotionDirector, showed the new method achieving the best trade-off between motion fidelity and scene preservation. Ablation experiments confirmed that all three components, user scene learning, the orthogonality loss, and homography guidance, are complementary and essential: removing any one of them degrades either the transferred motion or the integrity of the user’s scene.

Human evaluation proved even more striking. In a perceptual study with 72 participants recruited through online crowdsourcing, the method was preferred for camera motion fidelity in 90.45 percent of trials and for scene preservation in 70.31 percent of trials, with statistical analysis using generalized estimating equations confirming robust preferences across all criteria. A second, task-based interaction study with 12 participants compared three interface paradigms: pure text prompts using Google’s Flow with Veo3, preset-based motion panels using Veo2, and the reference-video-driven workflow. The reference-driven approach required an average of only 1.11 iterations and about 159 seconds per task, compared with nearly 9 minutes for text-based prompting, and it produced significantly lower cognitive load on every dimension of the NASA Task Load Index, along with dramatically higher satisfaction and preference scores.

The implications extend well beyond convenience. The researchers frame the work as a step toward democratizing cinematic production, noting that the expressive gap currently restricts access to advanced visual storytelling for small-scale industries, educators, and casual creators, a concern aligned with inclusive innovation goals. The system runs on a single A5000 GPU in roughly ten minutes per reference-video and image pair, comparable to existing personalization workflows, and its modular design means it could plug into stronger video backbones as they emerge, though adapting to modern architectures with unified spatiotemporal attention requires additional head-identification steps. Limitations remain: the homography guidance can be fooled by large moving foreground objects, textureless scenes provide too few keypoints, and the current interface offers no way to blend multiple reference motions or edit trajectories interactively. The team plans to address these gaps with multi-reference fusion, interactive trajectory editing, and broader studies with professional filmmakers. For now, the message is clear: the language of the camera, once the exclusive dialect of cinematographers, is becoming something anyone can speak simply by showing the machine what they mean.

Subject of Research: Zero-shot camera motion transfer from reference videos to static images using diffusion models

Article Title: Zero-shot personalized camera motion control for image-to-video synthesis

Article References: Guhan, P., Kothandaraman, D., Lee, G., Huang, T.-W., Su, G.-M., & Manocha, D. (2026). Zero-shot personalized camera motion control for image-to-video synthesis. Discover Artificial Intelligence, 6(1), Article 1377. https://doi.org/10.1007/s44163-026-02212-0

Image Credits: AI Generated

DOI: 10.1007/s44163-026-02212-0

Keywords: camera motion transfer, image-to-video synthesis, diffusion models, LoRA fine-tuning, homography, generative AI, video generation, human-computer interaction, creative tools, user study, CameraScore metric, zero-shot learning