Gen4P: Video Generators are 4D Perceivers

• Monocular full 4D reconstruction must recover scene geometry, depth, camera poses, and 3D trajectories while capturing static structure and motion across viewpoints and time.Source: §1 · p. 1
• Existing generative-prior approaches address 4D perception with task-specific methods and often require dense, frame-aligned supervision, limiting arbitrary spatiotemporal inputs.Source: §1 · p. 2
• Gen4P turns a pretrained video generator into a direct-regression 4D perceiver, using one encoder pass and shared scene tokens across queries instead of iterative denoising.Sharing scene tokens reduces encoding cost when making repeated 4D predictions.Source: §1 · p. 2
• A lightweight cross-attention decoder supports spatially flexible output sampling, while continuous-time queries accommodate variable-length videos and a frozen depth LoRA strengthens geometry without added trainable parameters.Source: §1 · p. 2
• The paper evaluates Gen4P across multiple 4D reconstruction tasks and reports qualitative results on in-the-wild videos.Source: §1 · p. 2

• Feed-forward geometry foundation models use pixel-aligned geometry representations, including pairwise point maps, spatial memory, recurrent scene states, and multi-view attention.Source: §2 · p. 2
• Tracking approaches range from depth lifting and iterative refinement to joint geometry-motion estimation; most 4D methods use frame-aligned grids, whereas D4RT uses on-demand queries [Xiao et al., 2024; Xiao et al., 2025; Zhang et al., 2026a].Source: §2 · p. 2
• Generative priors have been adapted to depth, point maps, scene flow, and tracking, with recent feed-forward video-generator adaptations targeting depth, dense 3D tracking, or unified perception [Hu et al., 2025; Xu et al., 2025; Zhu et al., 2026; Zhang et al., 2026b; Nam et al., 2026a; Wang et al., 2026b].Source: §2 · pp. 2–3

3 METHODS

• Gen4P uses on-demand queries to decode 4D attributes from a shared video representation, replacing iterative denoising with a single-pass feed-forward perceiver.The queried attributes include 3D tracking, point clouds, video depth, and camera parameters.Source: §3 · p. 3, §3.2 · p. 4

3.1 PRELIMINARIES: D4RT

• D4RT encodes a fixed-length video into shared scene features, then independently queries those features to predict 3D points and auxiliary 2D attributes.Each query specifies a pixel, source time, target time, and camera coordinate system.Source: §3.1 · pp. 3–4

3.2.1 VIDEO GENERATIVE ENCODER

• Low-noise features generally outperform higher-noise features on tracking EPE; Gen4P therefore uses clean video latents for feature extraction.The comparison includes σ = 0.05 versus σ = 0.35–0.5; the encoder uses final-block features before the velocity prediction head.Source: §3.2.1 · p. 4, Figure 3 · p. 4

3.2.2 QUERY DECODER

• Continuous-time queries replace fixed 48-frame temporal lookup tables with a parameterization that supports variable-length sequences in training and inference.Timestamps are represented by normalized position within the clip, independent of absolute sequence length.Source: §3.2.2 · pp. 4–5

3.3 PLUG-IN GEOMETRY PRIOR

• A frozen pretrained video-depth adapter supplies a geometry prior, while the task-specific 4D update and decoder are trained for 4D perception.The double-LoRA variant uses the same number of trainable parameters as 4D-LoRA alone and needs no additional dense prediction head.Source: §3.3 · p. 5

3.4 TRAINING OBJECTIVE

• Gen4P trains its predicted 4D attributes with a composite loss covering 3D and 2D position, confidence, visibility, displacement, and surface normals.Each listed attribute has a corresponding loss term.Source: §3.4 · p. 5

4.1 TRAINING SETUPS

• Training uses approximately 200K clips from 11 public datasets and compares a motion-prior variant with a frozen-depth-LoRA variant.Both variants have 0.70B trainable encoder parameters, with an 8-block cross-attention decoder containing 0.18B trainable parameters.Source: §4.1 · pp. 5–6

4.2 WORLD-COORDINATE TRACKING

• Gen4P outperforms prior methods by a clear margin on DR and PS, remains competitive on ADT and PO, and qualitatively preserves delicate structures during tracking.The reported tracking comparison highlights dynamic points, while in-the-wild visualizations show more plausible motion and better preservation of details such as thin legs and a front wheel.Source: Figure 4 · p. 6, Table 1 · p. 7, §4.2 · p. 7

4.3 VIDEO DEPTH ESTIMATION

• Video-depth evaluation targets three unseen datasets—Bonn, Sintel, and KITTI.The cited table reports results across these datasets.Source: §4.3 · p. 7, Table 2 · p. 7

4.4 CAMERA POSE ESTIMATION

• Ours-II achieves the lowest errors on all three Sintel metrics; both variants have the lowest relative translation error on TUM-Dynamics at 0.007 and remain competitive on ScanNetv2.The reported metrics are ATE, RPET, and RPER.Source: Table 3 · p. 8, §4.4 · p. 8

4.5 ANALYSIS

• Time-embedding ablations improve ADT while maintaining strong performance elsewhere; sequences extend to 105 frames without significant degradation, and AbsRel stays stable across query resolutions.Figure 5 compares depth across input resolutions and point tracking across sequence lengths, using 4RC as the baseline.Source: §4.5 · pp. 8–9, Figure 5 · p. 8
• On background regions, Gen4P exhibits less motion leakage and a clearer motion boundary than 4RC in the displacement-map comparison.The visualization mean-normalizes query-frame points and maps accumulated displacement over a shared range of [0, 0.5].Source: §4.5 · p. 9, Figure 6 · p. 9
• A wider LoRA does not guarantee better overall performance, while full DiT fine-tuning degrades several metrics despite using more trainable parameters.The compared adaptation strategies include variants that nearly double trainable parameters.Source: §4.5 · p. 9, Table 4 · p. 9
• The geometry prior improves performance across all reported metrics, and combining the 4D and depth LoRAs further improves overall performance.The comparison evaluates variants with and without the geometry prior.Source: §4.5 · p. 9, Table 4 · p. 9

5 CONCLUSION

• A single-step feed-forward perceiver uses pretrained video-generator priors to reconstruct full 4D attributes from monocular video.This differs from traditional multi-step denoising-based 4D perceivers.Source: §5 · p. 9
• On-demand decoding supports flexible spatial and temporal queries, while a frozen video-depth LoRA adds a geometry prior without extra trainable parameters.Source: §5 · p. 9
• Gen4P achieves state-of-the-art and flexible 4D reconstruction, with future improvement expected as video generative pretraining advances.The authors also report accurate capture of dynamic motion and intricate details in real-world videos.Source: §5 · pp. 9–10

AI USE STATEMENT

• AI tools assisted with grammar correction, language polishing, and figure and table layout adjustments.Source: AI USE STATEMENT · p. 11
• The authors reviewed and revised the AI-assisted text and checked it against the methods and experimental results.Source: AI USE STATEMENT · p. 11
• The authors assume full responsibility for the final manuscript, including its scientific claims and reported results.Source: AI USE STATEMENT · p. 11

REPRODUCIBILITY STATEMENT

• Sections 3 and C describe the model architecture, continuous-time query, plug-in geometry prior, training objective, and loss terms.Source: REPRODUCIBILITY STATEMENT · p. 11
• Section 4 and Appendices B.1–B.2 report training data, optimization settings, resolutions, sampling, preprocessing, query sampling, and ADT sequences excluded from training.Source: REPRODUCIBILITY STATEMENT · p. 11
• Appendices D.1–D.3 specify evaluation protocols, metrics, alignment, inference setups, and baseline configurations; datasets, Wan2.1, and the depth LoRA are publicly available.Source: REPRODUCIBILITY STATEMENT · p. 11

A.1 ADDITIONAL QUALITATIVE RESULTS

• Additional qualitative results include merged point clouds, tracking, camera-pose estimation, and further 3D tracking showcases.Source: §A.1 · p. 17

A.2 CONFIDENCE FILTERING

• Confidence filtering at a percentile as low as 2% suppresses flying points and produces cleaner foreground and background content.Raw tracking outputs are already visually clean but may contain a small number of flying points.Source: §A.2 · p. 17
• Figures 4 and 6 show methods without confidence-based suppression for fair comparisons.Source: §A.2 · p. 17

B.1 DATASETS

• Encoder frames are center-cropped to one of five aspect ratios and resized within 480 × 832, while query frames retain the crop and receive downsizing-only scale augmentation.No random content cropping, color jittering, or Gaussian blurring is applied.Source: §B.1 · p. 17

B.2 PREPROCESSING AND SAMPLING

• Raw videos are subsampled into 49-frame clips and encoded with Wan-VAE; training samples 21–49-frame query clips and at most 2,048 queries per clip.Forty percent of queries use matching target and camera times; the rest are randomly sampled.Source: §B.2 · pp. 17, 19
• ADT, DR, and PO 3D targets use mean-normalized points transformed to the camera coordinate system at tcam, while 2D position and visibility targets are defined in the observed ttgt frame.Cross-time visibility is unavailable directly from dataset annotations; the authors report that these 2D targets work well as auxiliary losses.Source: §B.2 · pp. 17, 19

B.3 REGARDING ARIA DIGITAL TWIN (ADT)

• ADT is included for comparable geometric supervision, with WorldTrack-ADT sequences excluded from training; adding ADT improves held-out ADT and PO tracking while other benchmarks stay consistent.One of the 50 evaluation clips has no visible frame-0 query points, leaving 49 valid results.Source: §B.3 · p. 20, Table A2 · p. 20, Table A3 · p. 20

C TRAINING OBJECTIVES

• Each query jointly predicts 3D position, confidence, 2D position, visibility, 3D displacement, and surface normal.The complete objective combines these six loss types with fixed weights across experiments.Source: C TRAINING OBJECTIVES · p. 20
• Each loss is averaged only over queries with valid ground-truth supervision for that loss type.The masked average uses the query count and a positive epsilon to prevent division by zero.Source: C TRAINING OBJECTIVES · p. 20
• Predicted and ground-truth 3D positions are independently normalized by mean absolute depth and signed-log transformed; confidence-weighted error plus a logarithmic regularizer prevents trivial confidence collapse.The confidence parameterization is c_i = 1 + exp(ĉ_i).Source: C TRAINING OBJECTIVES · pp. 20–21
• 2D positions use an element-wise Huber loss, visibility uses binary cross-entropy, displacement uses its camera-frame target, and normals use cosine distance after unit normalization.Visibility is positive only for physically visible points with positive depth inside the target image.Source: C TRAINING OBJECTIVES · p. 21

D.1 WORLD-COORDINATE TRACKING

• A 5 × 10^-3 m motion threshold is used for world-coordinate tracking: raising it to 1 × 10^-2 changes the average static ratio only from 56.82% to 57.29%, while 5 × 10^-2 misclassifies more static tracks.The static ratio on ADT rises from 67.23% to 74.30% at 5 × 10^-2; PStudio, which is fully dynamic, reaches 2.00%. Metrics use a single global median scale alignment over 64-frame tracks.Source: §D.1 · p. 21, Table A4 · p. 22

D.2 VIDEO DEPTH ESTIMATION

• Video-depth evaluation uses dataset-specific valid-depth ranges, aligns chunks through globally optimized scale-shift pairs on overlap, then aligns predictions to ground truth for metric computation.Sintel and Bonn use GT depths from 0 to 70 m; KITTI uses all valid GT depth pixels without a maximum-depth threshold.Source: §D.2 · pp. 22–23

D.3 CAMERA POSE ESTIMATION

• Camera pose estimation encodes overlapping chunks once, estimates relative poses from sparse queries in sliding three-frame windows, and refines within-chunk poses through a pose graph.The first pose is fixed during robust nonlinear least-squares refinement; chunks are then Sim(3)-aligned and fused into one trajectory.Source: §D.3 · pp. 24–25
• Chunk trajectories are aligned using Sim(3) transformations estimated from overlapping-frame orientations and centers, then fused by averaging aligned centers and projecting summed rotations onto SO(3).This yields a camera trajectory covering the complete input sequence.Source: §D.3 · p. 25