An arXiv paper introduces UniCaMo, a single framework for controlling both camera movement and object motion in image-and-text-to-video diffusion models.
Inputs: A reference image, a text prompt, sparse 3D object trajectories and a 3D camera trajectory.
Method: It builds motion-consistent input noise, combining noise propagated from the reference with spherical Gaussian noise for newly revealed regions, without reconstructing unseen scene geometry.
Setup: No architectural changes or dedicated control modules. It is adapted with lightweight LoRA fine-tuning on a low volume of training data.
Claims: The authors report state-of-the-art motion controllability and high visual quality on standard benchmarks, with minimal inference overhead. Qualitative videos are on the project page.
Motion goes in through the starting noise, not through new modules.
Sign in to comment.
TL;DR of the paper, generated automatically.
An arXiv paper introduces UniCaMo, a single framework for controlling both camera movement and object motion in image-and-text-to-video diffusion models.
Motion goes in through the starting noise, not through new modules.