PaperlayerView exact revision on arXiv

Source-linked AI summary

AdaViewPlanner: Adapting Video Diffusion Models for Viewpoint Planning in 4D Scenes

Yu Li, Menghan Xia, Gongye Liu, Jianhong Bai, Xintao Wang, Conglang Zhang, Yuxuan Lin, Ruihang Chu, Pengfei Wan, Yujiu Yang

arXiv:2510.10670v1cs.CV

TL;DR

The paper asks whether pretrained text-to-video models can support viewpoint planning for 4D scenes, a task for which manual camera design is tedious and specialized methods may struggle with open-world scenes. It adapts a T2V model in two stages to generate video and extract camera poses, and reports significantly better performance than prior approaches. The work presents this as a proof of concept for video-generation models in 4D interaction.

  • Problem

    Manual viewpoint design for 4D scenes is tedious and specialized, while existing methods can struggle with open-world generalization and text-based preference controls.

    Source: §1 · p. 2

  • Method

    A two-stage framework injects 4D scene representations into a pretrained T2V model, then uses generated video and the scene as conditions for camera-pose denoising.

    Source: §1 · p. 2, §5 · p. 10

  • Results

    The authors report that their method significantly outperforms prior approaches, with ablations verifying the effectiveness of its core designs.

    Source: §1 · p. 2, §5 · p. 10

  • Takeaways & Limitations

    The work offers a proof of concept for using video-generation models as world models for 4D interaction.

    Source: §1 · p. 2

    Unifying the two stages remains future work because their noise-timestep needs conflict, noisy video weakens camera cues, and real data with both skeletal and camera annotations is difficult to obtain.

    Source: §4.3 · p. 9

Abstract

from arXiv · show

Recent Text-to-Video (T2V) models have demonstrated powerful capability in visual simulation of real-world geometry and physical laws, indicating its potential as implicit world models. Inspired by this, we explore the feasibility of leveraging the video generation prior for viewpoint planning from given 4D scenes, since videos internally accompany dynamic scenes with natural viewpoints. To this end, we propose a two-stage paradigm to adapt pre-trained T2V models for viewpoint prediction, in a compatible manner. First, we inject the 4D scene representation into the pre-trained T2V model via an adaptive learning branch, where the 4D scene is viewpoint-agnostic and the conditional generated video embeds the viewpoints visually. Then, we formulate viewpoint extraction as a hybrid-condition guided camera extrinsic denoising process. Specifically, a camera extrinsic diffusion branch is further introduced onto the pre-trained T2V model, by taking the generated video and 4D scene as input. Experimental results show the superiority of our proposed method over existing competitors, and ablation studies validate the effectiveness of our key technical designs. To some extent, this work proves the potential of video generation models toward 4D interaction in real world.

1 INTRODUCTION

Automatic cinematography for 4D scenes is needed because manual camera design is tedious and specialized, while existing specialized methods can struggle with open-world scenes and preference controls. The paper adapts pretrained T2V models in two stages and reports outperforming existing competitors by a large margin.

Motivation

  • Automatic cinematography is sought because manually arranging camera shots for 4D content is tedious and requires specialized expertise.The target is camera viewpoints and movements for rendering dynamic 3D content as engaging video.Source: §1 · p. 2
  • Existing specialized methods can struggle to generalize to open-world scenes and lack preference controls such as text instructions.The paper proposes repurposing T2V models, whose training data and text-following ability are presented as advantages.Source: §1 · p. 2

Proposed approach

  • The method adapts pretrained T2V models through a two-stage process: inject a 4D scene representation, then extract camera viewpoints through guided camera-parameter denoising.Ground-truth camera poses can guide video generation; the camera diffusion branch takes the generated video and 4D scene as input.Source: §1 · p. 2

Results and contribution

  • Experiments report that the proposed method outperforms existing competitors by a large margin, and ablations validate its technical designs.The authors present adapting pretrained T2V models for 4D viewpoint planning as a proof of concept for video-generation models as world models.Source: §1 · p. 2

2 RELATED WORKS

Related work spans camera-trajectory planning from reference-based and character-driven methods to multimodal and text-conditioned generation. Human-video generation has also incorporated increasingly explicit 3D motion and scene representations.

Camera Planning

  • Camera-planning research includes reference-based data-driven methods, character-driven imitation and GAN approaches, multimodal dance-and-music controls, and text-conditioned trajectory generation.Examples include DanceCamera3D and Dancecamanimator for dance and music, and Director and Director3D for text-conditioned generation.Source: §2 · pp. 2–3

Synthesis

  • The related-work discussion situates camera planning and human-motion control as complementary lines of work addressing camera trajectories and explicit 3D structure, respectively.The paper’s approach is described as using normalized 4D human poses while the video model plans viewpoints and camera motion.Source: §2 · pp. 2–3

Human Motion Control for VDMs

  • Human-video control has progressed from 2D pose guidance to 3D motion conditions and methods that place characters in coherent 3D scene or world coordinates.Prior approaches either use fixed-viewpoint 2D pose sequences or require 4D human poses together with camera parameters; this work conditions on normalized 4D human poses.Source: §2 · p. 3

3 METHOD

The method uses pretrained video generation to design cinematic viewpoints from human motion, then directly extracts camera poses aligned with the motion’s coordinate system. It covers moving humans in 4D scenes and uses a three-branch model for pose extraction.

3 METHOD

  • The method focuses on moving humans in 4D scenes and formulates a camera trajectory from a sequence of 3D human joint positions.Each frame’s camera parameters comprise 3D translation and a 6D rotation representation that can be orthogonalized into a valid rotation matrix.Source: §3 · p. 3

3.1 STAGE I: MOTION-CONDITIONED CINEMATIC VIDEO GENERATION

  • Stage I conditions a pretrained video model on normalized human motion, using encoded motion tokens and spatial motion attention to generate video with learned cinematography.The model can receive explicit camera information with probability p during guided learning before autonomous camera design; the base video model is frozen while the new motion encoder and attention layers are trained.Source: §3.1 · p. 4

3.2 STAGE II: CAMERA POSE EXTRACTION

  • Direct estimation addresses post-processing for coordinate alignment and feature-matching failures caused by geometric and texture inconsistencies in AI-generated video.The cited issues include jittery or fragmented camera paths and failures in scene reconstruction.Source: §3.2 · p. 5
  • Stage II directly predicts absolute camera poses in the human coordinate system using video, camera, and motion branches in an MMDiT framework.It uses flow matching to transport noisy camera parameters toward clean ones; synthetic Unreal Engine data and GVHMR reconstruction provide paired motion and camera data in a common coordinate system.Source: §3.2 · pp. 4–5

4 EXPERIMENTAL RESULTS

Experiments report qualitative and quantitative advantages over baselines, then examine motion control, camera-generation design choices, and challenges to unifying the two stages.

4.1 EXPERIMENT SETTINGS

  • Both stages use a pretrained 1B Transformer T2V backbone; evaluation compares the method with E.T. and adapted DanceCamera3D, using rule-based metrics, MLLM-based evaluation, and a user study.The E.T. test set is refined to 500 samples, and the internal test set contains 240 samples.Source: §4.1 · pp. 5–6, §4.2 · p. 6

4.2 COMPARISON WITH STATE-OF-THE-ART METHODS

  • AdaViewPlanner adapts to scene styles and camera instructions, while varying random seeds produces different trajectories for the same human motion.The generated instruction-following trajectories include multi-view and multi-motion paths centered on the subject.Source: §4.2 · p. 6, Figure 4 · p. 7
  • Figure 5 contrasts AdaViewPlanner’s smoother, instruction-following, human-centered trajectories with E.T.’s simpler paths and DanceCam*’s collapsed, less diverse trajectory.The authors attribute DanceCam*’s collapse to difficulty converging on this task.Source: §4.2 · p. 6, Figure 5 · p. 7
  • Across both test sets, AdaViewPlanner outperforms all baselines, with more character-centered, smooth, diverse trajectories and stronger text consistency and cinematographic diversity.The reported measures include human visibility, camera jerk, shot diversity, text-camera consistency, and cinematographic style diversity.Source: §4.2 · pp. 6–7, Table 1 · p. 8

4.3 MORE ANALYSIS AND ABLATION STUDIES

  • Camera extraction degrades when Stage-I video and motion diverge; removing Stage-II motion conditioning yields smooth but misfocused paths, while relative-camera prediction lacks scale awareness.The ablations also report that Stage-I motion-consistent videos help improve Stage-II camera results.Source: §4.3 · pp. 7–8, Table 3 · p. 9
  • Their method matches Wan-2.1-14B base MTVCrafter on TikTok dance and significantly outperforms it on the general domain, evaluated under shared 4D motion conditions using WA-MPJPE and PA-MPJPE in millimeters.The authors attribute the general-domain result to training on broader pose distributions.Source: §4.3 · p. 8, Table 2 · p. 8
  • Unifying Stages I and II remains future work because their timestep needs conflict, noisy videos weaken pixel motion cues, and accurate camera annotations for real videos remain a bottleneck.Source: §4.3 · p. 9
  • Text-and-pose variants without the video model either fail to converge or collapse to a single trajectory, whereas the two-stage framework produces more diverse and reliable trajectories.Figure 7 compares training-loss curves for AdaViewPlanner, DanceCam*, and the variant without the video model.Source: §4.3 · p. 9, Figure 7 · p. 9

5 CONCLUSION

The conclusion presents a two-stage adaptation of pretrained T2V models for 4D viewpoint planning, reporting gains over prior approaches and effective core designs; the appendix maps supplementary material.

5 CONCLUSION

  • A two-stage framework uses an adaptive learning branch to inject 4D scene representations and a camera diffusion branch for hybrid-condition viewpoint denoising.Ground-truth viewpoints guide Stage I training.Source: §5 · p. 10
  • Experiments show significant gains over prior approaches, and ablations verify the effectiveness of the core technical designs.Source: §5 · p. 10

APPENDIX

  • Appendix A introduces the base text-to-video generation model, while B provides implementation details.The appendix lists six sections.Source: APPENDIX · p. 13
  • Appendix C–F cover evaluation metrics, MLLM evaluation details, additional analyses and visualizations, and editorial LLM use.Source: APPENDIX · p. 13

A INTRODUCTION OF THE BASE TEXT-TO-VIDEO GENERATION MODEL

The base video generator encodes videos into latents and uses transformer diffusion with joint spatiotemporal attention to generate coherent videos.

A INTRODUCTION OF THE BASE TEXT-TO-VIDEO GENERATION MODEL

B IMPLEMENTATION DETAILS

Implementation uses distinct sampling settings for Stage I and Stage II, and canonicalizes reconstructed human motion so Stage II estimates only camera extrinsics.

B IMPLEMENTATION DETAILS

  • Stage I uses 50 sampling steps, timestep shift 15, and CFG=5 for text and human motion, dropping motion conditioning for the last 10 steps.Motion is dropped late because it mainly influences mid-to-high noise regions; larger CFG values tend to introduce artifacts.Source: B IMPLEMENTATION DETAILS · p. 13
  • Stage II uses 50 sampling steps with timestep shift 1 and no classifier-free guidance.Source: B IMPLEMENTATION DETAILS · p. 13
  • GVHMR reconstructions are aligned to human coordinates and transformed into canonical space, with first-frame gravity set to negative y, frontal direction to positive z, and translation normalized to zero.Focal length is fixed at GVHMR’s nominal values, so Stage II estimates extrinsic parameters only.Source: B IMPLEMENTATION DETAILS · p. 14

C EVALUATION METRICS

Evaluation combines rule-based trajectory measures, MLLM judgments of high-level qualities, and researcher preferences, while noting limitations of learned text-camera evaluators.

C EVALUATION METRICS

  • Prior FCD and text-camera consistency scores rely on evaluators whose training-data limitations may bias scores out of distribution, and they omit camera–human motion interaction.The authors adapt and refine reference-free metrics from DanceCamera3D.Source: C EVALUATION METRICS · p. 14
  • Rule-based evaluation measures human visibility with HMR, smoothness with translational and rotational jerk, and shot diversity using camera distance and viewing angle.The geometric diversity measure distinguishes perspectives such as frontal and rear shots, unlike view-invariant projected human scale.Source: C EVALUATION METRICS · p. 14
  • MLLM evaluation uses orthographic trajectory visualizations to score text-camera consistency from 0 to 2 and quantify cinematographic-style diversity with entropy.The model assesses perspective, distance, and motion type.Source: C EVALUATION METRICS · p. 14
  • The user study samples 30 test instances per set and asks researchers to choose the best result using instruction following, cinematographic style, and motion coordination criteria.Those choices yield a user preference rate.Source: C EVALUATION METRICS · p. 14

D DETAILS OF MLLM-BASED EVALUATION

The evaluation addresses the lack of automated high-level trajectory metrics by using MLLMs to assess text–camera consistency and cinematographic style diversity. Three-view trajectory projections and marked start/end points and orientations provide spatial cues for the model.

D DETAILS OF MLLM-BASED EVALUATION

  • The evaluation targets text–camera consistency and cinematographic style diversity, dimensions for which the authors say automated high-level trajectory metrics are lacking.The authors use advanced MLLMs, including Gemini 2.5 Pro, for these assessments.Source: D DETAILS OF MLLM-BASED EVALUATION · p. 14
  • Three-view projections from top-down, front, and side views mark trajectory endpoints and orientations to help the MLLM reason about spatial relationships.The authors report that a 2D video alone makes those relationships difficult for the model to comprehend.Source: D DETAILS OF MLLM-BASED EVALUATION · p. 14
  • Figure 9 shows consistency examples in rows 1–2 and cinematographic-style-diversity examples in rows 3–4, with Gemini 2.5 Pro outputs at right.The middle column displays three-view projections; the left pairs text with 3D spatial visualizations.Source: Figure 9 · p. 15, D DETAILS OF MLLM-BASED EVALUATION · p. 15

E ADDITIONAL ANALYSES AND VISUALIZATION RESULTS

The section gives reasons for Stage II camera estimation and identifies problems with applying conventional estimation methods to AI-generated videos.

E.1 COMPARISON WITH 3D CAMERA MOTION ESTIMATION

  • Existing approaches that take videos or multiple images without explicit human-motion conditions can produce scale inconsistencies.The authors list this as one reason Stage II is needed.Source: §E.1 · p. 16
  • Conventional methods often recover only relative, normalized trajectories, making alignment with reference human poses computationally expensive to post-process.The authors also identify geometry and texture mismatches in AI-generated videos as a challenge for feature matching.Source: §E.1 · p. 17
  • Figure 10 compares camera estimation on AI-generated videos, illustrating reported risks of jitter, fragmented paths, and reconstruction failures.The authors attribute these problems to geometry and texture mismatches that degrade feature-matching algorithms.Source: Figure 10 · p. 15, §E.1 · p. 17

E.2 MORE VISUALIZATION RESULTS

Additional visualizations present pipeline ablations and generated trajectories, while the paper separately describes its restricted use of language models for editorial assistance.

E.2 MORE VISUALIZATION RESULTS

  • Figure 11 presents additional pipeline-design ablations, which the authors say validate the method's rationality.The passage does not report specific ablation values.Source: §E.2 · p. 17, Figure 11 · p. 18
  • Figure 12 illustrates generated trajectories for plausibility, diversity, and instruction consistency.Its caption describes diverse trajectories and the ability to follow camera text instructions.Source: §E.2 · p. 17, Figure 12 · p. 18
  • Figures 13 and 14 provide further complete results, described as cinematic, diverse, and high-quality camera trajectories.The passage does not specify quantitative measures for these qualities.Source: §E.2 · p. 17, Figure 13 · p. 19, Figure 14 · p. 20

F USE OF LARGE LANGUAGE MODELS

  • The authors say language models were used only for editing wording, clarity, and style, with each edited sentence reviewed and approved by co-authors.They state models were not used to generate research questions, design methods, interpret results, or draw conclusions.Source: F USE OF LARGE LANGUAGE MODELS · pp. 17–18
Loading 2510.10670v1…