Source-linked AI summary
OmniCam: Omni-Camera Trajectory Generation via Geometry-Grounded Pose Token Learning
Zhenyang Liu, Chenjie Cao, Yisu Zhang, Xuhui Zuo, Xiangyang Xue, Yanwei Fu, Tengfei Wang, Chunchao Guo
TL;DR
Language-guided camera trajectory generation must account for scene geometry and target framing, yet prescribed motion can intersect geometry or lose the intended target. OmniCam generates trajectories from panoramas and text using geometric, semantic, and target-anchor conditioning with hybrid pose tokens, and OmniCaT supplies synthetic training trajectories. On OmniCaT, OmniCam reduces ATE by 46.6% and collision by 65.8% relative to retrained GenDoP.
Problem
Generating camera trajectories from language requires accounting for scene geometry and target framing, since prescribed paths may intersect geometry or lose their intended target.
Source: §1 · pp. 1–2Method
Results
On OmniCaT, OmniCam reduces ATE by 46.6% and collision by 65.8% relative to retrained GenDoP.
Source: §5.2 · p. 7Takeaways & Limitations
The reported comparisons and ablations support these design choices within evaluated settings, while downstream studies examine their use in video generation and robotic perception.
Source: §1 · p. 2, §6 · p. 9The work is limited by estimated and incomplete geometry, planner-derived supervision, and static-scene observations.
Source: §6 · p. 9
Abstract
from arXiv · showhide
Camera trajectories control viewpoint changes in video generation, scene reconstruction, and robotic perception. Generating them from language requires both scene geometry and target-aware framing. We introduce OmniCam, an autoregressive model that generates camera pose sequences from a single panorama and textual trajectory descriptions. Its geometry-grounded pose token learning combines three components: a panoramic point-cloud encoder for omnidirectional geometric context; hybrid absolute-rotation and relative-translation tokenization with temporally consistent quaternion signs; and separate geometric and semantic conditioning streams with an explicit 3D target anchor. We also construct OmniCaT, containing 267,700 trajectories across four camera behaviors. On the reported OmniCaT evaluation, OmniCam reduces trajectory errors by 28--47% and collision rate by 65.8% relative to GenDoP retrained on OmniCaT. Against the best baseline for each metric, the ATE and collision reductions are 43.0% and 62.3%, respectively. Component ablations support the use of geometric and target-aware conditioning, while downstream experiments examine camera-controlled video generation and robotic active perception.
1. Introduction
OmniCam addresses language-guided camera trajectory generation that must account for scene geometry and target framing, since prescribed paths can intersect geometry or lose their target. It combines panoramic geometry, hybrid pose tokens, and separate geometric and semantic conditioning, and introduces a dataset and evaluations for these choices.
1. Introduction
- Prescribed camera paths may intersect scene geometry or lose their intended target, motivating scene-aware trajectory generation from a panorama and textual instruction.The task involves deciding where to move, how to orient, and what to attend to.Source: §1 · p. 1
- Quaternion sign alignment gives equivalent rotations consistent representatives, while absolute rotations avoid compounding rotation-quantization residuals and relative translations allow displacement error to accumulate.The point cloud reconstructed from estimated panoramic depth provides omnidirectional but partial geometry.Source: §1 · p. 2
- OmniCaT contains 267,700 trajectories across Target, Surrounding, Reconstruction, and Wandering behaviors, paired with hierarchical descriptions of task, target, and motion.Its supervision is generated on estimated geometry, so the learned model may inherit reconstruction errors and planner preferences.Source: §1 · p. 2
- OmniCam combines panoramic point-cloud geometry, hybrid pose tokenization, and separate geometric and semantic pathways with an explicit 3D target anchor.The geometry pathway does not directly receive text, and an attention-derived target centroid is injected through gated cross-attention.Source: §1 · p. 2
- On OmniCaT, the reported comparisons improve on retrained baselines, while component ablations and downstream evaluations examine the design and its use in video and robotic perception.The introduction lists panorama-conditioned generation, hybrid serialization, and dataset evaluation among the contributions.Source: §1 · p. 2
2. Related Work
Prior camera trajectory methods use varied inputs and representations, while existing datasets do not jointly provide omnidirectional geometry, diverse spatial behaviors, and hierarchical language guidance. OmniCam and OmniCaT address these gaps with panoramic point-cloud conditioning, target anchoring, and structured annotations.
Camera Trajectory Datasets
- Existing datasets lack the combination of omnidirectional scene geometry, diverse spatial behaviors, and hierarchical textual guidance that OmniCaT provides.MVImgNet [33], RealEstate10K [37], and DL3DV-10K [24] focus on simpler motions or static environments; CCD [18] and E.T. provide character-centric tracking data.Source: §2 · p. 2
Camera Trajectory Generation
- Camera trajectory methods span optimization-based planning [28], learning-based control [4], and newer methods with task-specific inputs and representations.OmniCam studies panoramic point-cloud conditioning and explicit target anchoring alongside absolute and relative pose serialization.Source: §2 · p. 2
Pose Representation Learning
- Sequential quaternion sign alignment reduces inconsistent discrete labels for equivalent rotations, but does not provide a globally continuous rotation parameterization or an injective finite quantizer.Earlier work studies rotation representations, including continuous 6D representations for single-frame regression [38].Source: §2 · p. 2
3. The OmniCaT Dataset
OmniCaT is a synthetic trajectory dataset built from geometry reconstructed from panoramas, with trajectories spanning four camera behaviors and hierarchical language annotations. Its construction combines geometric and semantic grounding, trajectory synthesis, and rendering with multimodal annotation.
3. The OmniCaT Dataset
- OmniCaT synthesizes camera trajectories on geometry reconstructed from panoramas to vary spatial behaviors, but feasibility against that geometry does not guarantee physical-scene collision avoidance.The model predicts from panorama and text at inference without constructing a navigation mesh or running graph search.Source: §3 · p. 3, §3.2 · p. 3
3.1. Dataset Construction
- The construction pipeline grounds panoramas into a point cloud, semantic landmarks, volumetric masks, and a navigation mesh, then synthesizes four behaviors and renders annotated trajectories.Camera behaviors are Target, Surrounding, Reconstruction, and Wandering; waypoints and orientations are refined with spline and spherical linear interpolation.Source: §3.1 · p. 3
3.2. Dataset Statistics
- 267,700 trajectories across 9,998 scenes and 21.7M rendered frames are annotated with semantic targets, directional bearings, and hierarchical textual descriptions.The behaviors comprise Surrounding (97,702), Target (90,065), Reconstruction (43,316), and Wandering (36,617) trajectories.Source: §3.2 · p. 3
4. Method
OmniCam represents camera poses with hybrid absolute and relative tokens and conditions an autoregressive decoder on geometric, semantic, and target-anchor information. The method separately describes its encoding tradeoffs, conditioning pathways, spatial grounding, and constrained decoding.
4.1. Hybrid Pose Tokenization
- Absolute quaternions with sequential sign alignment and relative translation increments are quantized into nine tokens per frame, balancing rotation-error behavior against translation range and accumulated displacement error.Each frame uses four rotation, three translation, and two intrinsics tokens; finite binning remains lossy.Source: §4.1 · p. 4
4.2. Decoupled Dual-Branch Conditioning
- A semantic branch processes caption and panorama, while a text-agnostic point-cloud branch supplies structural descriptors for trajectory generation.The geometric pathway receives no direct text input; this separates input routing without assuming statistically independent representations.Source: §4.2 · pp. 4–5
4.3. Spatial Grounding via Resampling and Target Centroid Injection
- Semantic resampling uses spatial and textual guidance, geometric resampling compresses point-cloud descriptors, and an attention-derived 3D target centroid provides an explicit spatial anchor.The centroid is encoded into auxiliary target memory, which is masked when caption dropout removes the target object tag.Source: §4.3 · p. 5
4.4. Format-Constrained Autoregressive Decoding
- The autoregressive Transformer generates pose tokens from multimodal conditioning, masks EOS within each nine-token frame group, and trains with weighted token cross-entropy plus centroid localization loss.Token masking enforces frame-boundary termination; it does not guarantee collision avoidance.Source: Figure 3 · p. 4, §4.4 · p. 5
5. Experiments
The experiments compare OmniCam with retrained and released-checkpoint baselines on trajectory quality, collision, visibility, and downstream tasks. Results favor OmniCam on OmniCaT and support geometric and target-aware components, while transfer and physical-robot evidence have stated scope limits.
5.1. Experimental Setup
- The evaluation uses disjoint OmniCaT splits and reports trajectory errors, reconstructed-mesh collision, target visibility, and angular deviation; reference errors measure agreement with generated supervision.The setup notes that multiple distinct trajectories can satisfy an instruction, while out-of-domain baseline rows use differing character-centric conditioning.Source: §5.1 · pp. 5, 7
5.2. Main Results
- On OmniCaT, OmniCam reduces ATE, FDE, RPE-R, and RPE-T by 46.6%, 41.1%, 28.0%, and 28.9% versus retrained GenDoP; collision falls 65.8%.Against the best baseline for ATE and collision, the reductions are 43.0% and 62.3%, respectively.Source: Table 1 · p. 6, §5.2 · p. 7
- OmniCam’s reported DataDoP results favor it over GenDoP trained on DataDoP, but transfer interpretation requires input-adaptation, data-overlap, and metric-protocol checks.The same section reports lower ATE and FDE across all four behavior categories and qualitative alignment with navigable regions, target viewpoints, and smoother transitions.Source: §5.2 · p. 8
5.3. Ablation Studies
- Removing the semantic branch lowers visibility from 52.0% to 11.5%, while removing geometry raises collision from 10.4% to 45.8%.These ablations support both pathways, but branch removal also changes capacity and available information.Source: §5.3 · p. 8
- Point-cloud, RGB-D, and RGB-only variants have collision rates of 10.4%, 22.6%, and 38.7%; removing target cross-attention lowers visibility and worsens angular deviation.Hybrid pose encoding also obtains lower RPE-R and RPE-T than the all-absolute and all-relative variants; removing sign alignment raises RPE-R to 1.847.Source: §5.3 · p. 8
5.4. Downstream Application Validation
- With WorldStereo, OmniCam trajectories improve the reported CLIP score by 14.1% and consecutive-frame SSIM by 15.4% relative to GenDoP.The real-robot demonstration illustrates feasibility in shown cases, not systematic sim-to-real performance; the success-rate aggregation needs verification.Source: §5.4 · p. 8
6. Conclusion
The conclusion presents OmniCam and OmniCaT as a geometry-grounded trajectory-generation approach supported across evaluated behaviors. The accompanying analyses describe the planning and training pipeline while bounding conclusions by synthetic supervision, estimated geometry, and finite evaluations.
6. Conclusion
- OmniCam combines panoramic point-cloud conditioning, hybrid pose serialization, and 3D target anchoring; conclusions remain bounded by estimated geometry, planner-derived supervision, and static-scene observations.The conclusion identifies dynamic scenes and closed-loop planning as future work.Source: §6 · pp. 8–9
A.1. Geometric Initialization and Refinement
- Geometry initialization uses 42 perspective views for depth coverage, aligns depth with LSMR, and filters sky, depth discontinuities, and NavMesh artifacts.The refined navigation graph samples NavMesh points at 0.05 m spacing and connects nearby points subject to height coherence.Source: §A.1 · p. 11
A.2. Trajectory Planning Heuristics
- Planning samples 72 target-path candidates, restricts surround and wander routes to NavMesh reachability, and smooths paths and orientations with behavior-specific rules.Reconstruction paths use reduced smoothing to preserve viewpoint transitions for reconstruction coverage.Source: §A.2 · p. 11
B. Training and Inference Details
- Training uses bf16 for approximately 100 effective epochs on eight H20 GPUs; reported timing includes depth estimation through decoding and does not establish real-time operation.The reported inference procedure uses top-k truncation at k = 10 and format-constrained decoding.Source: B. Training and Inference Details · pp. 11–12
C. Encoder Architecture Details
- The encoder routes appearance and text through a semantic pathway and point-cloud features through a geometry pathway, then fuses their outputs in the decoder.Removing the geometry branch raises collision from 10.4% to 45.8%, though this does not isolate routing from capacity or input-information changes.Source: C. Encoder Architecture Details · p. 12
D. Per-Behavior Decomposition
- OmniCam has lower ATE than retrained GenDoP for Wander (1.150 vs. 2.421), Target (0.677 vs. 1.088), Surround (1.258 vs. 1.932), and Reconstruct (0.385 vs. 1.055).The reported Target and Reconstruct reductions are 37.8% and 63.5%; these comparisons do not isolate causes of the gains.Source: D. Per-Behavior Decomposition · p. 12
E. Scaling Analysis
- The evaluated data-volume and model-size comparisons show trends only within their finite range, not a scaling law or absence of representational bottlenecks.The conclusion also distinguishes quaternion sign consistency from lossy quantization and notes error accumulation with relative encoding.Source: §6 · p. 9, E. Scaling Analysis · p. 12
E.1. Data Scaling
The scaling experiments report improving trajectory metrics across four nominal data scales and three tested model sizes. The reported comparisons show lower ATE and collision and higher visibility at larger scales.
E.1. Data Scaling
- ATE falls from 1.483 to 0.868, collision from 19.8% to 10.4%, and visibility rises from 34.6% to 52.0% across the four data scales.The scales are labeled 27K, 53K, 107K, and 267K.Source: §E.1 · p. 12
E.2. Model Scaling
- Small, Base, and Large models have approximately 120M, 500M, and 1B trainable parameters, respectively, with other hyperparameters held constant as architecture dimensions change.The compared configurations use hidden dimensions 512, 1024, and 1536; 8, 8, and 16 heads; and 8, 12, and 24 layers.Source: §E.2 · p. 12
- ATE decreases from 1.127 to 0.743, collision from 13.8% to 8.7%, and visibility increases from 44.6% to 56.3% across the tested model sizes.These observations support gains at the tested sizes but do not establish which representational choices explain the trend.Source: §E.2 · p. 12
F. Failure Case Analysis
The failure analysis identifies limitations in geometry perception, target grounding, and the static-scene assumption. These cases bound the reported system's use in environments with noisy geometry, ambiguous or occluded targets, reflective surfaces, or dynamic objects.
Failure Case Analysis
- Noisy or incomplete depth reconstructions near thin structures or in large open environments can make the geometric branch miss collision boundaries or hallucinate obstacles, producing physically implausible paths.Ambiguous target phrases may also ground the attention-aggregated centroid to the wrong instance, especially without spatial descriptors or when objects look alike from the panorama.Source: F. Failure Case Analysis · p. 12
- A static panoramic observation cannot react to moving objects, potentially yielding collisions with dynamic obstacles or failure to track moving targets.The paper identifies video-conditioned or closed-loop settings as future work.Source: F. Failure Case Analysis · p. 12
- Reflective or transparent surfaces can make visual features ambiguous, while heavy target occlusion can shift the estimated centroid toward irrelevant visible regions.These cases can yield unreliable target-aware conditioning, erratic orientation, or grounding away from the true target.Source: F. Failure Case Analysis · p. 13
- The authors caution that collision metrics are not deployment guarantees and that physical deployment requires validation against the actual environment.Errors in inferred geometry or open-loop trajectories can produce unsafe camera motions.Source: F. Failure Case Analysis · p. 13
G. Attention Visualization and Centroid Analysis
Attention visualizations show concentration near referenced scene regions, while the caption-dropout ablation reports a tradeoff among trajectory metrics. The visualizations complement, but do not quantify, the grounding ablation.
G. Attention Visualization and Centroid Analysis
- The visualization presents the input panorama, reconstructed point cloud, and attention projected onto scene points.Point-level aggregation is specified conceptually in Equation (2), while implementation details require verification.Source: G. Attention Visualization and… · p. 13
- The visualizations show attention concentrated near referenced regions when projected onto the reconstructed point cloud, but do not quantify localization accuracy or establish that attention alone causes improved visibility.The target centroid is described as an explicit spatial anchor for the trajectory.Source: G. Attention Visualization and… · p. 13
H. Caption Dropout Ablation
- 52.0% visibility and 10.9° angular deviation with five-variant dropout compare with 38.7% and 18.4° without dropout, while no dropout has slightly lower ATE and collision.No-dropout ATE and collision are 0.856 and 10.1%, versus 0.868 and 10.4% for the five-variant scheme; attribution to instruction-format overfitting requires separate evaluation.Source: H. Caption Dropout Ablation · p. 13
I. Theoretical Analysis of Hybrid Pose Tokenization
The analysis bounds representation-error tradeoffs in hybrid pose tokenization, compares OmniCam with a privileged heuristic planner, and presents qualitative trajectory and downstream examples.
I. Theoretical Analysis of Hybrid Pose Tokenization
- Under the stated normalization and decoder assumptions, B equal component bins give a conditional rotation-error bound of 2 arcsin(2/B).The bound follows from a component-dequantization error bound and quaternion normalization; it does not apply without those decoder and range assumptions.Source: §I · pp. 13, 15
- Relative-rotation reconstruction has worst-case error bound min{π, (t −1)ϵ} from exact initialization and per-step errors ≤ϵ; absolute decoding avoids integrating earlier rotation-quantization residuals.This is a worst-case bound, not an expected growth rate; residuals can cancel, and autoregressive absolute-token prediction errors need not be independent.Source: §I · p. 15
- Quaternion sign alignment makes the token sequence independent of input q versus −q representatives under deterministic conventions, but finite quantization remains many-to-one.At an exactly π inter-frame rotation, the inner-product sign rule needs an additional deterministic tie rule.Source: §I · p. 15
J. Heuristic Planner Oracle Comparison
- 5.2% collision and 58.7% visibility for the heuristic planner compare with OmniCam’s 10.4% and 52.0%; reported runtimes are approximately 16.5s versus 0.84s.The planner uses reconstructed geometry, navigation meshes, semantic masks, and target annotations; this is a privileged-information comparison on estimated geometry, not physical ground truth.Source: Table 8 · p. 16, J. Heuristic Planner Oracle Comparison · p. 16
I. Theoretical Analysis of Hybrid Pose Tokenization
- Relative translation improves local resolution when D ≪ L, while long sequences can accumulate translation error; rotating local frames also require accounting for orientation errors.The trajectory-level benefit of the hybrid representation is a design tradeoff requiring empirical evaluation, not a guarantee of universally minimal pose error.Source: §I · p. 16
K. More Qualitative Results
- Selected comparisons show trajectory-placement and orientation differences across scenes, while downstream examples pair panoramas with generated video frames and reconstructions.The examples are qualitative and do not establish dataset-wide collision or smoothness guarantees, nor quantify reconstruction accuracy.Source: K. More Qualitative Results · p. 16
L. Downstream Application Validation
Downstream experiments evaluate OmniCam trajectories as controls for video generation and robotic active perception, alongside qualitative examples across scene types.
Additional Qualitative Results
- Qualitative trajectory comparisons span indoor, outdoor, and stylized scenes; downstream examples show panoramas, selected generated video frames, and reconstructions with and without OmniCam trajectories.The figures illustrate selected outputs rather than quantitative reconstruction accuracy or completeness.Source: Figure 8 · p. 17, Table 9 · p. 18, Figure 9 · p. 19
Camera-Controlled Video Generation
- 14.1% CLIP score, 15.4% consecutive-frame SSIM, 18.0% aesthetic proxy, and 9.2% known-region fidelity improvements over GenDoP are reported.The same video-generation model uses different trajectory sources; results describe the trajectory-plus-video pipeline and do not isolate framing, path length, speed, or geometric consistency as causes.Source: L. Downstream Application Validation · p. 16, Table 10 · p. 18
Robotic Manipulation (Active Perception)
- 72.5% grasp success is reported for OmniCam, versus 48.0% for GenDoP and 56.3% for a fixed camera, but the fixed-camera aggregation is unresolved.The record describes 200 episodes across 50 tabletop scenes; physical-robot examples illustrate feasibility in shown cases, while systematic real-world claims require repeated trials and matched controls.Source: L. Downstream Application Validation · pp. 16, 20, Table 11 · p. 20
M. Statistical Confidence Analysis
The confidence analysis estimates evaluation-scene variability for OmniCam and retrained GenDoP using bootstrap resampling.
M. Statistical Confidence Analysis
- Table 12 reports non-overlapping 95% confidence intervals from 1,000 bootstrap resamples of OmniCaT evaluation scenes for OmniCam and retrained GenDoP.These intervals characterize variation across evaluation scenes, not training seeds, and do not cover all other experiments; GenDoP is not the best baseline for every main-table metric.Source: M. Statistical Confidence Analysis · p. 20, Table 12 · p. 21