PaperlayerView exact revision on arXiv

Source-linked AI summary

Dynamic Camera Poses and Where to Find Them

Chris Rockwell, Joseph Tung, Tsung-Yi Lin, Ming-Yu Liu, David F. Fouhey, Chen-Hsuan Lin

arXiv:2504.17788v1cs.CV

TL;DR

Suitable dynamic Internet videos are scarce, and estimating their camera poses is difficult. DynPose-100K combines video filtering with tracking, dynamic masking, and structure-from-motion; experiments report pose-error reductions of up to 90% and improved filtering precision. The dataset offers a diverse collection of camera-annotated videos for research applications.

  • Problem

    Most Internet videos are unsuitable for camera-pose estimation, and dynamic objects and changing appearances make pose estimation challenging.

    Source: §3 · p. 3

  • Method

    DynPose-100K combines specialist models and a generalist VLM for filtering with point tracking, dynamic masking, and structure-from-motion for pose annotation.

    Source: §1 · p. 2, §3.2 · p. 4

  • Results

    Up to 90% lower pose-estimation error across metrics and settings; filtering achieves 0.78 test precision, which no baseline reaches at any recall except reprojection error at 0.02 recall.

    Source: §1 · p. 2, §4.1 · p. 5

  • Takeaways & Limitations

    DynPose-100K provides diverse camera-annotated dynamic videos relevant to camera-controlled generation, view synthesis, and robotics applications.

    Source: §1 · p. 2

    The selection criteria exclude videos with severe zoom effects, abrupt shot changes, ambiguous reference frames, or insufficient static correspondences.

    Source: §3.1 · p. 3

Abstract

from arXiv · show

Annotating camera poses on dynamic Internet videos at scale is critical for advancing fields like realistic video generation and simulation. However, collecting such a dataset is difficult, as most Internet videos are unsuitable for pose estimation. Furthermore, annotating dynamic Internet videos present significant challenges even for state-of-theart methods. In this paper, we introduce DynPose-100K, a large-scale dataset of dynamic Internet videos annotated with camera poses. Our collection pipeline addresses filtering using a carefully combined set of task-specific and generalist models. For pose estimation, we combine the latest techniques of point tracking, dynamic masking, and structure-from-motion to achieve improvements over the state-of-the-art approaches. Our analysis and experiments demonstrate that DynPose-100K is both large-scale and diverse across several key attributes, opening up avenues for advancements in various downstream applications.

1. Introduction

Dynamic Internet videos could support camera-aware generation, view synthesis, and robotics, but suitable videos are scarce and pose estimation remains difficult amid motion and changing appearances. DynPose-100K combines broad filtering with dynamic-pose annotation to produce a large, diverse collection.

1. Introduction

  • Dynamic Internet videos could support camera-controlled generation, 4D scenes, view synthesis, and robotics applications.The paper identifies imitation learning and realistic simulation environments as robotics applications.Source: §1 · p. 2
  • Existing datasets use synthetic videos or restricted real-world domains, but these approaches constrain scale or data diversity.Synthetic datasets are typically under 500 videos, while restricted capture setups make structure-from-motion easier but limit collected data.Source: §1 · p. 2
  • DynPose-100K contains diverse dynamic Internet videos annotated with camera information and targeted video lengths.The dataset is curated from Panda-70M using filtering and pose-estimation pipelines.Source: §1 · p. 2
  • Pose estimation error falls across metrics and settings by as much as 90%, while filtering selects videos with higher precision than alternatives.The filtering and pose-estimation pipelines combine specialist and generalist models with tracking, masking, and structure-from-motion components.Source: §1 · p. 2

2. Related Work

Prior camera-pose datasets rely on synthetic content or restricted real-world capture settings that simplify estimation but constrain the data. This work instead uses diverse Internet videos and combines masking, long-term tracking, and classical structure-from-motion.

2. Related Work

  • Synthetic datasets offer ground-truth poses but are often small, while real-video datasets use restricted domains and capture setups that constrain their data.The passage describes synthetic datasets as typically under 500 videos and notes that special capture setups simplify structure-from-motion.Source: §2 · p. 2
  • Dynamic Internet video challenges structure-from-motion because dynamic correspondences cannot support bundle adjustment and limit static correspondences.Prior approaches mask dynamics using learned prediction or motion and semantic cues.Source: §2 · p. 2
  • The approach builds on BootsTAP tracking and Theia-SfM, using classical structure-from-motion for precision on longer videos.The passage contrasts this design with concurrent approaches focused on small dynamics and circular camera motion, or learned pose methods for dynamic scenes.Source: §2 · p. 3

3. DynPose-100K: Dataset Curation

DynPose-100K addresses the scarcity of suitable Internet videos and the difficulty of estimating poses in dynamic scenes. Its curation combines eligibility criteria, specialist filters and a generalist VLM, while pose estimation links dynamic masking, point tracking, and global bundle adjustment.

3. DynPose-100K: Dataset Curation

  • DynPose-100K targets diverse dynamic Internet videos with camera annotations, addressing unsuitable source videos and difficult pose estimation in dynamic scenes.The dataset curation section frames these as its two primary challenges.Source: §3 · p. 3

3.1. Candidate Selection Criteria

  • Only 9% of a randomly selected 1,000-video Panda-70M test set met the criteria for dynamic camera-pose estimation.Videos should be real and sufficiently high-quality, feasible for pose prediction, and contain both a moving camera and dynamic scene.Source: §3.1 · p. 3

3.2. Candidate Video Selection

  • Specialist models filter frequent rejection cases, while a prompted VLM detects a broader variety of issues; their combined use is effective.The VLM addresses eight rejection questions, including ambiguous references, blurred backgrounds, distortion, static scenes, cartoons, post-processing, and children.Source: §3.2 · pp. 3–4

3.3. Dynamic Camera Pose Estimation

  • Sliding-window point tracking supplies dense, long-term correspondences, dynamic masks remove non-static tracks, and global bundle adjustment uses the remaining tracks.Masking combines semantic, interaction, and motion cues with mask propagation; tracking is designed to maintain correspondences across occlusion and drift.Source: Figure 3 · p. 4, §3.3 · pp. 4–5

4. DynPose-100K: Dataset Analysis

The section evaluates whether filtering can find pose-suitable Internet videos, then characterizes DynPose-100K’s scale and the diversity of its content and dynamics.

4.1. Filtering Evaluation on Panda-Test

  • On 1,000 manually classified Panda-Test videos, 90 (9%) were suitable; evaluation prioritized precision while also tracking recall.Source: §4.1 · p. 5
  • 0.78 precision at DynPose-100K’s collection threshold; no baseline reaches it at any recall except reprojection error at 0.02 recall.Source: Figure 5 · p. 5, §4.1 · p. 5
  • +Flow, tracking, masking, focal, distortion, and VLM each improve filtering; VLM adds a large boost despite weak standalone performance.Source: §4.1 · p. 5, Figure 5 · p. 5

4.2. Dataset statistics

  • Frequent nouns span subjects, objects, and settings, while verbs reflect varied actions, providing evidence of diverse video content.Source: §4.2 · p. 5, Figure 4 · p. 6
  • DynPose-100K has the most videos among diverse Internet-video datasets; diverse-content alternatives CamCo and B-Timer have fewer videos and are private.Source: §4.2 · p. 5, Table 1 · p. 6
  • Most DynPose-100K videos span 4 to 10 seconds, a duration described as rich in dynamics yet long enough for substantial camera motion.Source: §4.2 · p. 6, Figure 6 · p. 6
  • Dynamic-object apparent sizes range from small to large; near-full-frame dynamic objects are filtered out because they make pose estimation infeasible.Source: §4.2 · p. 6, Figure 6 · p. 6

5. Evaluation on Camera Pose Estimation

Across a synthetic ground-truth benchmark and a real-video correspondence benchmark, Ours achieves lower pose error than alternatives. The evaluation reports reductions on both Lightspeed and Panda-Test.

5. Evaluation on Camera Pose Estimation

  • Evaluation pairs Lightspeed’s ground-truth poses with Panda-Test’s 10K annotated correspondences and Sampson-error scoring for Internet videos without ground-truth poses.Panda-Test reprojection error is the square root of Sampson error, averaged for each video.Source: §5 · p. 6, §5.2 · p. 7

5.1. Pose Evaluation on Lightspeed

  • Lightspeed is a photorealistic dynamic-pose benchmark with ground-truth poses, and its racing sequences include challenging indoor and outdoor scenes and lighting.The qualitative comparison shows static methods struggling and Ours producing smooth, accurate trajectories.Source: §5.1 · p. 6, Figure 7 · p. 7

5.1. Pose Evaluation on Lightspeed; 5.2. Pose Evaluation on Panda-Test

  • On Lightspeed, Ours cuts trajectory error 50% across all sequences and 90% on easy sequences versus all methods; on Panda-Test, mean error falls over 35% versus all methods on 52 jointly successful sequences.The reported Lightspeed comparison is in Table 2; Panda-Test uses the 52 sequences all methods register.Source: Table 2 · p. 7, §5.1 · p. 7, Table 3 · p. 8

5.2. Pose Evaluation on Panda-Test

  • On Panda-Test, static methods are inaccurate or fail to register many videos, while Ours handles difficult large-dynamics and varied-appearance cases better.The table reports Ours outperforming across metrics; qualitative examples show several alternatives struggling with large dynamic regions or varied appearance and lighting.Source: §5.2 · pp. 7–8, Table 3 · p. 8, Figure 8 · p. 8
  • DynPose-100K fine-tuning of DUSt3R yields lower mean error and similar-or-better accuracy than MonST3R synthetic data using 2K videos and 140K frames versus 1.3M frames.This is the Table 4 comparison on Panda-Test.Source: Table 4 · p. 8

6. Conclusion

The paper presents DynPose-100K, a large-scale dynamic Internet-video dataset annotated with camera poses, curated through filtering and pose-estimation pipelines. It also reports that fine-tuning DUSt3R on DynPose-100K lowers mean error versus MonST3R synthetic data and introduces Lightspeed for ground-truth benchmarking.

6. Conclusion

  • DynPose-100K is a large-scale dataset of dynamic Internet videos annotated with camera poses.The paper describes it as curated through video-filtering and camera-pose-estimation pipelines validated by experiments.Source: §6 · p. 8
  • The paper introduces DynPose-100K, a dataset of dynamic Internet video annotated with camera poses, and reports that fine-tuning DUSt3R on it produces lower mean error than MonST3R synthetic data (Table 4).Source: §6 · p. 8
  • Lightspeed supplements DynPose-100K with a high-quality synthetic dataset for ground-truth pose benchmarking.Source: §6 · p. 8

Appendix A. DynPose-100K: Dataset Curation

The appendix details video-selection criteria and filters, then describes the dynamic masking and tracking components used in camera-pose estimation. Filtering scores combine available filters, while mask components target different sources of motion.

A.1. Candidate Selection Criteria

  • Candidate videos are screened for realism and quality, pose feasibility, and dynamic cameras and scenes.Excluded cases include cartoons or poor-quality footage, zoom or shot changes and ambiguous reference frames, and static cameras or scenes.Source: §A.1 · p. 13

A.2. Candidate Video Selection

  • Videos receive the average of available filter scores, with thresholds tuned on a non-overlapping 1K Panda-Val set; a final threshold of 0.910 selects 137K videos, about 4.3% of 3.2M.Source: §A.2 · pp. 13, 15
  • Specialist filters assess acceptability, distortion, focal length, dynamic-mask size, flow, and tracks; the VLM rejects a clip if any of its eight exclusion questions is answered yes.The filter stages are ordered to reduce compute cost, with the VLM applied last to minimize API costs.Source: §A.2 · pp. 13, 15
  • The focal-length filter rejects clips when the 90th-to-10th percentile spread exceeds 40% of the mean, a one-second change exceeds 20%, or the 80th percentile exceeds 1400.The three sub-scores are averaged.Source: §A.2 · p. 13
  • To reduce filtering cost across about 3.2M videos, motion and shot-change checks use sampled tracks and flow; the abbreviated tracking filter runs at 3 fps for 30 frames.A shot-change score is zero if more than 50% of tracks disappear across one frame; dynamics score is zero when median tracks move less than about 5% of frame size.Source: §A.2 · pp. 13–15

A.3. Dynamic Camera Pose Estimation

  • Dynamic masking combines semantics for common dynamic classes, interaction for manipulated objects, and motion for objects moving without human contact, then propagates their masks smoothly.The components can complement one another to produce more complete masks.Source: §A.3 · p. 15, Figure 10 · p. 16
  • The pipeline runs segmentation every six frames and uses SAM2 to propagate combined masks six frames forward, producing masks over half-second intervals at 12 fps.Propagation is several times faster than masking and substantially speeds the overall process.Source: §A.3 · p. 15
  • DynPose-100K tracks a 42-by-42 point grid at 256-by-256 resolution, with a five-frame stride and 30-frame tracking window; short clips are padded with empty frames when needed.The selected tracking settings reflect the collection’s efficiency goal, and equal-length sequences let BootsTAP run faster than shortened ones.Source: §A.3 · p. 15

B.1. Filtering Evaluation on Panda-Test

The filtering evaluation manually labels 1,000 Panda-Test videos, of which 90 (9%) are suitable. The appendix explains how Table 5 presents precision-recall results and identifies the operating threshold.

B.1. Filtering Evaluation on Panda-Test

  • Manual review labels 90 of 1,000 randomly selected Panda-Test videos (9%) as suitable for dynamic camera-pose estimation.Videos are checked for realism, quality and ethics, pose estimability, dynamic camera, and dynamic scene.Source: §B.1 · p. 16
  • Table 5’s left panel smooths the precision-recall curve by taking the maximum precision among points with recall at least as high as each point.This makes precision non-increasing as recall rises.Source: §B.1 · p. 16
  • The right panel reports unsmoothed precision values, and recall 0.40 is the operating threshold used for DynPose-100K.Source: §B.1 · p. 16

C.1. Pose Evaluation on Lightspeed

The Lightspeed evaluation runs methods on full-rate 24 fps clips and uses trajectory registration and alignment rules to measure pose accuracy. The reported tracking setup achieves competitive final results despite shorter temporal coverage than the other datasets.

C.1. Pose Evaluation on Lightspeed

  • All methods run on Lightspeed clips extracted at the full 24 fps because the clips are short.Additional sequences are 2560 × 1440 pixels.Source: §C.1 · p. 16
  • A sequence registering less than 80% of frames counts as failed; failed trajectories receive random translations before alignment for ATE.The method succeeds on all Lightspeed sequences, while repeating SfM on failed sequences for other methods made minimal or no difference.Source: §C.1 · p. 16
  • Our tracking produces competitive final results with 40-frame windows spanning 1.67 seconds, shorter than the 2.5 seconds used on DynPose-100K and Panda-Test.Forward-and-backward tracking had minimal impact compared with forward-only tracking.Source: §C.1 · p. 17

C.2. Pose Evaluation on Panda-Test

On Panda-Test, the method produces superior pose estimates and runs faster than ParticleSfM in the reported A40 timing comparison. The section also details correspondence validation, filtering behavior, dataset examples, and the use of reprojection error to identify higher-quality poses.

C.2. Pose Evaluation on Panda-Test

  • 11,866 correspondence pairs were collected, and 10,210 (86.0%) had agreement between human annotations and SuperPoint+LightGlue matches.Matches had to fall within 10 pixels on 720p images of both human-annotated points; human matches provide coarse accuracy, while SuperPoint+LightGlue provides precision.Source: §C.2 · p. 17, Figure 13 · p. 19
  • 11 minutes versus ParticleSfM’s 44 minutes: our method averages less runtime on the same 10-video A40 sample while producing superior pose estimates.Our tracking, masking, and SfM take about 3.3, 4.2, and 3.5 minutes, respectively; ParticleSfM’s components take 26, 2, and 16 minutes.Source: §C.2 · pp. 17–18

Additional Results

D.1. Filtering Evaluation on Panda-Test

  • Combined filters are typically, but not always, effective; individual filters can miss zoom-ins, shot changes, and static cameras.DynPose-100K videos must achieve a minimum average filter score of 0.91.Source: §D.1 · p. 19

D.2. Dataset Overview

  • DynPose-100K examples span varied lighting, movement, and apparent dynamic-object sizes, while the paper describes their pose annotations as high quality.Source: §D.2 · p. 19

D.3. Reprojection Error Analysis

  • Filtering videos by low reprojection error significantly reduces pose error and can produce high-quality dataset subsets.The paper reports that Table 6 evaluates pose accuracy before and after this filtering on Panda-Test.Source: §D.3 · p. 19, Table 6 · p. 19

D.4. Dataset Rotation Analysis

DynPose-100K includes substantial horizontal camera rotation, and the additional pose comparisons show the method handling challenging dynamic scenes on Lightspeed. The appendix also presents further pose results on both Lightspeed and Panda-Test.

D.4. Dataset Rotation Analysis

  • Final horizontal rotations often exceed 30 degrees, with sequential total horizontal rotation exceeding 75 degrees.Figure 17 plots final horizontal and vertical rotations and sequential sums of their absolute values.Source: §D.4 · p. 19, Figure 17 · p. 22
  • Sample DynPose-100K videos pair challenging trajectories and dynamics with pose annotations.The dataset samples are described as diverse.Source: Figure 15 · p. 21

Appendix E. Additional Pose Results

  • Appendix E presents additional pose results on both Lightspeed and Panda-Test.Source: Appendix E · p. 22

E.1. Pose Evaluation on Lightspeed

  • On Lightspeed, Ours handles scenes with a dynamic object static relative to the camera and with a spinning object kicking up dust, where alternatives struggle.The comparisons include cases with moving cameras and difficult lighting.Source: Figure 16 · p. 21, §E.1 · p. 22

E.2. Pose Evaluation on Panda-Test

On Panda-Test, Ours handles challenging dynamic objects and appearance variation, and the ablations compare tracking, masking, and bundle-adjustment choices. BootsTAP performs better than CoTracker, every masking component contributes to final performance, and Theia-SfM has lower mean error than GLOMAP.

E.2. Pose Evaluation on Panda-Test

  • BootsTAP produces better pose accuracy than CoTracker on Panda-Test.The comparison is reported in Table 7.Source: §E.2 · p. 22
  • Removing any of the four masking components reduces average final performance.The ablation is reported in Table 8 and is also apparent in Figure 10.Source: §E.2 · p. 22, Table 8 · p. 22
  • GLOMAP offers competitive precision, but Theia-SfM has lower mean error on Panda-Test.The paper notes GLOMAP failures on Internet video tend to be more severe, which is most clearly reflected in mean error.Source: Table 9 · p. 22, §E.2 · p. 22
  • Ours best handles large dynamic objects and appearance or lighting variation on Panda-Test; additional examples show reasonable trajectories across varied videos.Source: §E.2 · p. 22, Figure 19 · p. 23, Figure 18 · p. 23
Loading 2504.17788v1…