Source-linked AI summary
GimbalDiffusion: Gravity-Aware Camera Control for Video Generation
Frédéric Fortier-Chouinard, Yannick Hold-Geoffroy, Valentin Deschaintre, Matheus Gadelha, Jean-François Lalonde
TL;DR
Text-to-video models lack precise control over extreme camera rotations, and common relative representations do not provide an absolute gravity reference. GimbalDiffusion grounds camera poses in gravity-aligned coordinates and uses panoramic video to cover extreme viewpoints; it outperforms dynamic baselines on absolute orientation metrics while reporting similar FID and FVD on par with UCPE and AC3D.
Problem
Text-to-video camera control remains limited for extreme rotations, while relative or ambiguous trajectory representations lack precise absolute geometric control.
Source: §1 · p. 2Method
GimbalDiffusion grounds camera poses in a gravity-aligned absolute coordinate system and uses rectified 360° videos to create trajectories across possible viewpoints.
Source: §1 · p. 2, §3 · p. 5, §3.2 · p. 7Results
GimbalDiffusion outperforms all dynamic video-generation baselines on PitchErr and GravityErr and nearly all on RotErr, with FID similar to other baselines and FVD on par with UCPE and AC3D.
Source: §4.5 · p. 13Takeaways & Limitations
The pipeline covers full-sphere viewpoints, including extreme pitch, large roll, and full 360° trajectories previously beyond generative video models.
Source: §5 · p. 15On-the-fly camera sampling is restricted to rotations and retains source-video translations; generated videos can also exhibit visual artifacts.
Source: §5 · p. 15
Abstract
from arXiv · showhide
Recent progress in text-to-video generation has achieved remarkable realism, yet fine-grained control over camera motion and orientation remains elusive, especially with extreme trajectories (e.g., a 180-degree turnaround, or looking directly up or down). Existing approaches typically encode camera trajectories using relative or ambiguous representations, limiting precise geometric control and offering limited support for large rotations. We introduce GimbalDiffusion, a framework that enables camera control grounded in physical-world coordinates, using gravity as a global reference. Instead of describing motion relative to previous frames, our method defines camera trajectories in an absolute coordinate system, allowing accurate, interpretable control over camera parameters. Using panoramic 360-degree videos for training, we cover the full sphere of possible viewpoints, including combinations of extreme pitch and roll that are out-of-distribution of conventional video data. To improve camera control, we introduce null-pitch conditioning, a strategy that prevents the model from overriding camera specifications in the presence of conflicting prompt content (e.g., generating grass while the camera points toward the sky). Finally, we propose new benchmarks to evaluate gravity-aware camera-controlled video generation, assessing models' ability to generate extreme camera angles and quantify their input prompt entanglement.
1 Introduction
Text-to-video models offer limited fine-grained camera control, especially for extreme rotations, and common relative representations lack a gravity-based absolute reference. GimbalDiffusion uses panoramic video, null-pitch conditioning, and new benchmarks to address these gaps.
1 Introduction
- Prior trajectory-conditioned methods often use ambiguous relative camera representations and limited-motion training data, restricting precise absolute control and extreme-rotation generalization.Relative-to-first-frame extrinsics do not provide an absolute orientation reference such as gravity.Source: §1 · p. 2
- Panoramic 360° video enables training trajectories that cover the sphere, including rare extreme viewpoints beyond conventional forward-facing paths.The method samples camera crops over the sphere, including barrel rolls, extreme high- and low-angle perspectives, and looping 360° rotations.Source: §1 · pp. 2–3
- Null-pitch conditioning captions forward-facing crops rather than the actual camera view to address conflicts between prompt content and camera orientation.For example, a downward-facing camera should render grass even when the prompt also mentions a blue sky.Source: §1 · pp. 2–3
- In the backflip example, GimbalDiffusion follows the requested trajectory and retains the first-frame content, while UCPE and GEN3C fail.The described trajectory looks up, backward, down, and then returns to its starting position.Source: Figure 2 · p. 3
- Two benchmarks assess extreme-angle video generation and entanglement between input prompts and camera orientation.The extreme-angle benchmark rebalances SpatialVID-HQ pitch samples; the entanglement benchmark annotates PolyHaven panorama crops.Source: §1 · p. 3
2 Related Work
Related work spans video generation and editing controls, camera-control methods, panorama generation, and camera-pose and gravity estimation. The paper identifies unresolved limits in absolute orientation control, extreme rotations, and panorama-based alternatives.
Video generation
Video editing
- Video-editing controls use edge maps, depth, sketches, or 3D geometry for structure, and illumination-aware guidance for lighting.These controls target changes to content, viewpoint, or lighting after synthesis.Source: §2 · pp. 3–4
Camera control methods
- Camera-control methods range from absolute-pose image generation to relative-trajectory video control, but several rely on reference-frame estimates or lack gravity-aligned orientation.UCPE encodes a latitude-up map but requires complex data generation and cannot handle extreme camera angles; reference-image approaches such as MotionCanvas depend on depth and pose estimation quality.Source: §2 · p. 4
Panorama video generation
- Panorama video generation can support gravity-aligned crops, but discards pixels, may crop out prompted concepts, and does not provide translation control in current text-conditioned methods.The cited panorama methods include.Source: §2 · p. 4
Camera pose estimation
- Camera-pose estimation spans classical structure-from-motion and learning-based methods that infer pose alongside scene geometry or dense correspondences.Examples include COLMAP, VGGT, DUSt3R, MASt3R, MegaSAM, and ViPE.Source: §2 · pp. 4–5
Gravity estimation
- Gravity estimation approaches may require auxiliary sensors or scene semantics, while existing gravity-annotated datasets can have limited environmental diversity.The paper instead calibrates PanoVid using its up vector and SfM-estimated intrinsics and extrinsics for a consistent absolute reference.Source: §2 · p. 5
3 Method
GimbalDiffusion grounds camera control in an absolute, gravity-aligned coordinate system, defining pitch and roll against a shared reference while leaving yaw unconstrained. Its pipeline represents camera rays, creates varied paths from panoramic videos, and uses null-pitch captions to strengthen adherence to camera conditioning.
3 Method
- GimbalDiffusion targets precise control over full 360° trajectories, extreme pitch, and large roll using an absolute, gravity-aligned reference.In this shared reference, pitch and roll are well defined, while yaw remains unconstrained.Source: §3 · p. 5
3.1 Representing cameras
- UCPE injects camera-ray transformations through trainable attention and adds latitude and up-vector information, while Plücker rays represent each pixel as a 6D ray.The Plücker ray direction is computed from the camera rotation, intrinsics, and pixel coordinates.Source: §3.1 · pp. 6–7
3.2 Creating varied camera paths from 360◦videos
- Panoramic poses are aligned to gravity using estimated up vectors, then combined with sampled rotations to create camera poses in a shared frame.The transformation removes yaw from the initial sampled pose, so sequences begin with null yaw.Source: §3.2 · p. 7
- Random initial pitch, roll, and yaw plus sequential rotations generate varied paths, while sampled and interpolated fields of view broaden viewpoint coverage.Field of view keyframes range from 35° to 100°, with sampling slightly biased toward the poles.Source: §3.2 · p. 8, Figure 4 · p. 8
3.3 Null-pitch conditioning
- Null-pitch conditioning captions an upright, forward-looking view while using the randomly rotated crops for diffusion or flow training loss.The captions describe the full scene, helping the model follow camera conditioning when prompt content conflicts with the specified view.Source: §3.3 · p. 9
- A look-down caption is appended 50% of the time and used as a negative prompt to suppress unwanted objects such as tripods.The strategy was most useful when fully finetuning, which tended to overfit to unwanted training-set artifacts.Source: §3.3 · p. 9
4 Experiments
The experiments evaluate GimbalDiffusion’s training and baseline configurations, introduce benchmarks for extreme camera control and prompt-camera entanglement, and report quantitative and qualitative results (p50).
4.1 Training data
- Training uses 4,798 PanoVid videos with high-quality camera poses after filtering out 38% of the original 7,797 videos.Each training iteration samples a 49-frame camera trajectory and captions six evenly spaced frames.Source: §4.1 · pp. 9–10
4.2 Models and baselines
- The comparison includes camera-control baselines, and GimbalDiffusion is trained with multiple camera-conditioning strategies and backbones; UCPE uses a less diverse auxiliary dataset.GEN3C is excluded from GimbalDiffusion training because it does not support dynamic scenes.Source: §4.2 · pp. 10–11
4.3 Video camera control benchmark
- SpatialVID-extreme uses 138 clips selected for uniform pitch coverage and adds randomly sampled roll trajectories from -40° to 40° by warping video.The benchmark starts from 371K SpatialVID-HQ clips; the resulting pitch and roll diversity supplies prompts and reference videos for FID and FVD.Source: §4.3 · p. 11
4.5 Results
- GimbalDiffusion outperforms all baselines on PitchErr and GravityErr, nearly all dynamic methods on RotErr, and is competitive on FID and FVD.Baselines initialized with PreciseCam improve FVD at the cost of worse camera-orientation metrics; GimbalDiffusion also shows a slight CLIP-score drop.Source: Table 1 · p. 12, §4.5 · p. 13
4.4 Metrics
- Camera control is evaluated with absolute pitch and gravity errors, relative rotation and translation errors, CLIP alignment, and FID/FVD visual-quality metrics.Roll error is not computed directly because it is unreliable near the poles; deviations are reflected in gravity error.Source: §4.4 · p. 12
4.5 Results
- Qualitative examples show that baselines struggle with extreme angles, while GimbalDiffusion aligns scenes such as ceilings with an upward-facing camera.Figure 5 overlays the input absolute camera angle; the method’s outputs align with the overlay, including sky at zenith and horizon at the equator line.Source: §4.5 · p. 13, Figure 5 · p. 13
- Removing absolute rotations increases pitch and gravity error 3.67× and 3.36×, while removing null-pitch conditioning raises pitch error 37% and 72% in two variants.The relative-camera ablation omits absolute extreme pitch and roll information from the model.Source: §4.5 · p. 14
4.6 Prompt-camera entanglement evaluation
- Null-pitch conditioning lowers pitch error across 20 scenes, especially at extreme pitches, and improves CLIP similarity toward the ground-truth curve.The benchmark tests 380 prompt-and-pitch pairs, spanning -90° to +90° in 10° increments.Source: §4.6 · p. 14, Figure 6 · p. 15
5 Discussion
GimbalDiffusion grounds camera poses in gravity-aligned absolute coordinates and uses panoramic training data to cover extreme viewpoints; the authors also identify constraints and visual artifacts (p39, p40, p43).
5 Discussion
- The method uses gravity-aligned absolute camera poses and panoramic video to cover extreme pitch, large roll, and full 360° rotation trajectories.Null-pitch conditioning is a data-generation step intended to enhance camera-conditioning fidelity when prompts contain conflicting cues.Source: §5 · p. 15
- On-the-fly camera sampling is limited to rotations and must retain source-video translations; generated videos can also contain visual artifacts.The authors suggest extending augmentation to translations and expect video-generation progress to mitigate artifacts.Source: §5 · p. 15
- The authors frame the work as a foundation for camera control under extreme rotations, with the aim of helping creators convey visual ideas more precisely and quickly.This is the paper’s stated intended benefit, not an evaluated outcome in these passages.Source: §5 · p. 15