Source-linked AI summary
Relational Geometry Forcing for World Action Models
Yuan Xu, Yixiang Chen, Qisen Ma, Jiabing Yang, Peiyan Li, Kai Wang, Jianhua Yang, Jianlou Si, Jun Huang, Jing Liu, Nianfeng Liu, Yan Huang, Liang Wang
TL;DR
WAMs lack direct geometric supervision despite geometry's importance for precise manipulation, and pointwise geometric alignment can degrade action learning. RGF aligns pairwise relations from a frozen 3D foundation model with video and action features. It reports competitive manipulation performance, real-world generalization, and consistent gains across WAM paradigms.
Problem
WAMs lack direct scene-geometry supervision, while pointwise alignment degrades action learning because dense geometric targets and compact action features encode different content.
Source: §1 · pp. 1–2Method
RGF transfers frozen 3D geometric priors by aligning intra-frame, cross-view, and action-temporal feature relations without explicit reconstruction or inference-time auxiliary components.
Source: §1 · p. 2Results
Takeaways & Limitations
Relational alignment provides a compatible interface for geometric supervision across heterogeneous WAM features while preserving the original video–action inference pathway.
Source: §5 · p. 7The current formulation focuses on pairwise cosine relations and leaves explicit metric scale and contact structure unexplored.
Source: §5 · p. 8
Abstract
from arXiv · showhide
World Action Models (WAMs) have shown strong potential for robot manipulation by jointly learning visual dynamics and executable actions. However, their video and action objectives do not explicitly constrain the 3D geometry that is crucial for precise manipulation. Recently, geometry-aware WAMs introduce depth or 4D prediction branches, whereas representation alignment methods supervise intermediate WAM features with pretrained 3D foundation models. Yet we find that such pointwise alignment benefits the video features but degrades action learning, since dense geometric feature targets overly restrict compact action features. To address this limitation, we propose Relational Geometry Forcing (RGF), which transfers geometric priors from a frozen 3D foundation model by aligning pairwise feature relations instead of pointwise features. Specifically, RGF applies three complementary relational objectives, where intra-frame and cross-view relations transfer scene geometry and multi-view correspondence to the video features, while action-temporal relations couple action dynamics to geometric scene evolution. Applied only during training, RGF requires no explicit geometry branch or extra inference module. Extensive experiments in simulation and the real world demonstrate the effectiveness of Relational Geometry Forcing, showing consistent gains across diverse manipulation scenarios and WAM paradigms.
1 Introduction
WAMs predict future video and actions but lack direct geometric supervision, motivating ways to transfer 3D priors for precise manipulation. RGF matches pairwise feature relations rather than pointwise targets, and the introduction reports competitive performance and generalization across simulated and real-world settings.
1 Introduction
- WAMs predict future RGB frames and action chunks, but their usual objectives provide no direct scene-geometry supervision.Recent geometry-aware variants add depth or 4D prediction branches.Source: §1 · p. 1
- Pointwise alignment helps video features but degrades action learning because dense scene targets and compact action features encode different content.The figure contrasts the pointwise approach with RGF's relational alignment, which benefits both video and action experts.Source: Figure 1 · p. 1, §1 · p. 2
- RGF transfers frozen 3D geometric priors by aligning pairwise WAM feature relations, without explicit geometric reconstruction.Intra-frame and cross-view objectives supervise video features; an action-temporal objective couples action dynamics to geometric scene evolution, with auxiliary components used only during training.Source: §1 · p. 2
- Experiments show competitive performance across single-arm and bimanual tasks, generalization to unseen real-world conditions, and consistent gains across WAM paradigms.The real-world conditions include unseen objects, background distractors, and height changes.Source: §1 · p. 2
2 Related Work
Prior WAMs add geometric prediction branches or use pointwise feature alignment, while robot-learning approaches often explicitly encode or reconstruct geometry. RGF instead aligns feature relations for video and action experts, retaining the original inference pathway.
World Action Models
- WAM approaches include multi-view RGB-D prediction, auxiliary depth prediction, and training-time 4D experts; RGF instead regularizes representations without a depth or 4D predictor.Fast-WAM uses video co-training without inference-time future generation.Source: §2 · p. 2
Representation Alignment
- Representation-alignment methods transfer priors through pointwise feature matching, whereas RGF aligns intra-frame, cross-view, and action-temporal relations in dual-expert WAMs.This avoids direct correspondence between compact action features and dense geometric targets.Source: §2 · p. 2
Geometry-Aware Robot Learning
- The WAM predicts future multi-view frames and an action chunk while the frozen 3D teacher encodes the same frames; auxiliary components are removed at inference.The relational objectives supervise intermediate video and action features.Source: Figure 3 · p. 3
- Many robot-learning methods explicitly encode or reconstruct geometry; RGF uses a frozen 3D foundation model for training-time supervision and keeps the original inference pathway.The cited approaches include voxels, multi-view features, 3D feature fields, point clouds, and 3D occupancy prediction.Source: §2 · p. 3
3 Method
RGF transfers geometric priors from a frozen 3D foundation model to WAM video and action features through pairwise relation alignment. Its three relational objectives supervise within-view geometry, cross-view correspondence, and action-temporal scene evolution.
3.1 Preliminaries
- The WAM jointly models future multi-view observations and action chunks, using video and action diffusion transformers trained with flow-matching losses.A frozen VGGT processes synchronized views to provide spatially resolved video targets and spatially pooled scene targets for action supervision (Wang et al. 2025).Source: §3.1 · pp. 3–4
3.2 Relational Geometry Forcing
- RGF aligns pairwise relations between WAM features and geometric targets rather than regressing features directly, accommodating representations with different roles.Video tokens encode local visual regions, while action tokens summarize control transitions.Source: §3.2 · p. 4
- Intra-frame and cross-view objectives transfer VGGT’s spatial-region relations and cross-view similarity patterns to video features.The cross-view objective matches relations for pairs of camera views; intra-frame relations alone do not capture correspondences between different projections of the same scene.Source: §3.2 · p. 4
- The action-temporal objective matches pairwise relations in action features to temporal similarity relations among the corresponding multi-view scenes.Action steps are temporally paired with scene representations, but supervision is applied between time-step pairs rather than regressing action tokens to scene vectors.Source: §3.2 · p. 4
3.3 Training Objective and Inference
- The full training objective combines the three relational terms with the original WAM objectives, while RGF components are removed at inference.Inference uses only RGB observations and language instructions, without geometric inputs or additional modules; the approach applies across several video–action prediction paradigms.Source: §3.3 · p. 4
4 Experiments
The experiments evaluate RGF across simulation, geometry-transfer analyses, ablations, qualitative comparisons, and real-world manipulation. Results report gains across benchmark settings, while analyses show how relational objectives affect transfer and learned geometric relations.
4.2 Main Results
- RGF-WAM achieves the highest average LIBERO success rate among compared methods, with gains over Fast-WAM on Spatial (+1.2), Goal (+1.6), and Long (+1.4), while both reach 100% on Object.On RoboTwin 2.0, it has the highest randomized and average success rates, surpassing MECo-WAM by 0.09 percentage points on average; versus Fast-WAM, clean performance rises from 91.88% to 92.92% and randomized performance from 91.78% to 92.50%.Source: §4.2 · p. 5
4.3 Analysis of Geometry Transfer
- Relational alignment improves both DINOv2- and VGGT-supervised variants, and gains persist across Uncond, Joint, and IDM WAM variants at +1.05, +0.60, and +0.55, respectively.Relational DINOv2 reaches 98.35%, exceeding video-only pointwise VGGT at 97.90%; with relational alignment fixed, VGGT improves over DINOv2 from 98.35% to 98.65%.Source: §4.3 · pp. 5–6
4.4 Ablation Studies
- The full combination of intra-frame, cross-view, and action-temporal objectives yields +1.05 versus +0.40 for the best two-objective variant, while individual objectives remain close to baseline.The two top two-objective variants include cross-view supervision; combining intra-frame with action-temporal yields −0.25, whereas cross-view with action-temporal yields +0.40. Huber scores 98.65% versus 98.05% for KL.Source: §4.4 · p. 6, Table 4 · p. 6
4.5 Qualitative Analysis
- RGF video features recover smoother within-view geometric similarity and place the highest-similarity patch near the ground-truth cross-view correspondence, unlike unaligned features.Figure 4 compares intra-frame similarity maps with ground-truth 3D distances and cross-view maps with marked correspondences.Source: §4.5 · p. 6, Figure 4 · p. 6
- For a rollout with nonuniform motion, RGF recovers the progressively narrowing scene-evolution pattern, whereas the unaligned WAM produces a nearly uniform-width action-temporal band.The rollout starts slowly and accelerates, so early frames remain similar over a longer interval than later frames.Source: Figure 4 · p. 6, §4.5 · p. 7
4.6 Real-World Experiments
- The real-world evaluation uses a Franka Research 3 arm with three cameras, four basic tasks, and three Bowl-to-Rack generalization variants, with 50 demonstrations per task and 20 evaluation rollouts per setting.The variants test unseen object color, background distractors, and altered placement heights, none present during training.Source: §4.6 · p. 7
- On real-world tasks, RGF-WAM has the highest average success rate, beats Fast-WAM by 8.6 percentage points, and is best or tied-best in six of seven settings.Against Fast-WAM, gains are +5 for unseen color, +5 for distractors, and +20 for altered placement heights; both methods score 85% on the original Bowl-to-Rack task.Source: §4.6 · p. 7, Table 5 · p. 7
5 Conclusion
The conclusion presents RGF as a way to transfer geometry to WAM features through pairwise relations rather than direct feature regression. Experiments support its use across simulation and real-world manipulation, while the current formulation leaves metric scale and contact structure unmodeled.
5 Conclusion
- RGF transfers geometric priors through pairwise feature relations, combining intra-frame, cross-view, and action-temporal objectives without explicit reconstruction or changes to the video–action inference pathway.The paper reports competitive manipulation performance, real-world generalization, and consistent gains across diverse WAM paradigms.Source: §5 · p. 7
- The current formulation focuses on pairwise cosine relations and leaves explicit metric scale and contact structure unexplored.Future work may incorporate these cues and longer-horizon dynamics while retaining training-time relational supervision.Source: §5 · p. 8
Appendix
- The appendix provides additional alignment-baseline, objective, training, architecture, hyperparameter, compute, benchmark, ablation, and rollout details.It includes suite-level LIBERO results, task-level RoboTwin results, and complete real-world rollout visualizations.Source: Appendix · p. 10
A Method Details
The method details describe pointwise feature alignment as a control and specify how RGF relations are constructed and used during training. RGF training caches frozen VGGT targets and removes its auxiliary components at inference.
A Method Details
- The method-details section formalizes pointwise alignment before describing relational-objective construction and the training workflow.The pointwise objective uses cosine distance, with video geometric targets taken from the corresponding spatial locations.Source: A Method Details · p. 10, §A.1 · p. 10
A.1 Pointwise Alignment
- Pointwise alignment resamples VGGT patch features to the WAM grid for video and pairs pooled per-view geometric targets with same-step action features.The video-only control disables the action pointwise term; when enabled, the video and action pointwise terms each have weight 0.5.Source: §A.1 · p. 10
A.3 Training Procedure
- Training precomputes frozen VGGT features, extracts layer-15 video and action features, constructs relations, and updates the WAM and projection heads with RGF and flow-matching losses.RGF is enabled throughout fine-tuning without a warm-up stage; cached targets, projection heads, and relation computations are removed at inference.Source: §A.2 · p. 10, §A.3 · p. 10
A.2 Relational Objectives
- RGF constructs intra-frame, cross-view, and action-temporal relations, pairing 32 action steps with pooled descriptors from consecutive observations independently of nine-frame video subsampling.It masks padded action steps and self-relations, averages over valid entries, and weights the three objectives equally.Source: §A.2 · p. 10
B Implementation Details
The implementation uses a shared pretrained video-diffusion backbone for the video and action experts, with separate projection heads for RGF. It evaluates three WAM paradigms under that shared backbone.
B.1 Model Architecture
- All experts initialize from Wan2.2-TI2V-5B; the action expert uses linearly interpolated video-DiT weights, and RGF uses separate two-layer projection heads removed at inference.The video and action heads map 3072→2048 and 1024→2048, respectively.Source: §B.1 · p. 11
- Uncond predicts actions from current visual context, Joint jointly denoises future video and actions, and IDM conditions action prediction on predicted future video representations.Source: §B.1 · p. 11
B.2 Training Hyperparameters
The section reports training settings and compute use alongside detailed LIBERO suite results. Relational alignment improves the reported average across all three WAM paradigms.
B.2 Training Hyperparameters
- Training uses AdamW at 1 × 10−4 learning rate, with a 5% linear warm-up, cosine schedule, and 10 epochs for LIBERO versus 5 for RoboTwin 2.0.Inference uses 10 denoising steps and classifier-free guidance scale 1.0.Source: §B.2 · p. 11
B.3 Compute Usage
- LIBERO runs require approximately 16 hours on 8 NVIDIA H20 GPUs; RoboTwin 2.0 runs require approximately 65 hours on 32 NVIDIA H20 GPUs.Source: §B.3 · p. 11
C.1 Geometry Transfer by Suite
- 98.35% for relational DINOv2 exceeds video-only pointwise VGGT, while Uncond action pointwise VGGT lowers the average from 97.90% to 97.55%.Relational alignment improves Goal and Long over the Uncond unaligned baseline.Source: §C.1 · p. 11
- 99.10% (Joint, from 98.50%) and 98.55% (IDM, from 98.00%) are the relational-alignment averages; Uncond reaches 98.65%.For IDM, relational alignment improves Spatial, Object, and Long while remaining close on Goal.Source: §C.1 · p. 11
C.2 Objective Ablations by Suite
The suite-level ablations find the strongest and most balanced result from combining all three relational objectives, while KL divergence gives a lower average than Huber. The compared RoboTwin methods span several policy families and geometry-aware approaches.
C.2 Objective Ablations by Suite
- Combining all three objectives produces the strongest, most balanced result, improving Spatial, Goal, and Long while retaining 100% on Object.Individual objectives remain near baseline; cross-view with action-temporal supervision is the best two-objective configuration at 98.00%.Source: §C.2 · p. 11
- 98.05% average with KL is lower than Huber, despite Spatial rising from 99.4% to 99.8%; Goal and Long each fall by 1.4 points.The authors report that preserving relation magnitudes with Huber provides a better balance across suites.Source: §C.2 · p. 11
D.1 Compared Baselines
- The comparisons include a pretrained VLA, pretrained WAMs, direct-action Fast-WAM, and geometry-aware WAMs; MECo-WAM is omitted because a complete 50-task breakdown is unavailable.RGF-WAM shares the Fast-WAM backbone and applies relational supervision without explicit geometry prediction.Source: §D.1 · p. 11
D.2 Task-Level Analysis
On RoboTwin 2.0, RGF-WAM has the highest randomized average among compared methods and best or tied-best results on many tasks; the layer ablation favors L15.
D.2 Task-Level Analysis
- RGF-WAM achieves the highest randomized average among compared methods and the best or tied-best results on 18 randomized and 23 clean tasks.Its gains span short-horizon interactions, fine-grained contact operations, and longer multi-stage manipulation, while performance remains uneven across tasks.Source: §D.2 · pp. 11–12
E Extra Ablation Studies
- The extra ablations use Uncond WAM, VGGT supervision, all three relational objectives, and Huber loss unless stated otherwise.They examine the supervised WAM layer and overall relational loss weight.Source: E Extra Ablation Studies · p. 12
E.1 Alignment Layer
- The middle layer L15 provides the strongest average result, outperforming L8 and L22 on Spatial, Goal, and Long while retaining 100% on Object.L8 still improves over the unaligned baseline; the passage suggests later layers may reshape injected relations, while deeper features may be less compatible with external supervision.Source: §E.1 · p. 12
E.2 RGF Loss Weight
In the LIBERO loss-weight ablation, λRGF=0.5 gives the strongest reported average among the stated weights; rollout visualizations show successful real-world task executions.
E.2 RGF Loss Weight
- 98.65% average at λRGF=0.5, up from 98.40%; λRGF=1.0 lowers the average to 97.55%.The reported gains at 0.5 include Spatial and Long; the authors use 0.5 as the default balance.Source: §E.2 · pp. 12, 14, Table A6 · p. 14
- A weight of 0.25 already improves over the unaligned baseline, while excessive relational supervision can interfere with the original video and action objectives.The largest degradation at λRGF=1.0 is on Long.Source: §E.2 · pp. 12, 14
F Real-World Rollout Visualizations
- The real-world visualizations show successful rollouts for four tasks, spanning single-object transfer, precise placement, and multi-stage manipulation.Each row shows eight uniformly sampled frames ordered left to right from one complete rollout.Source: F Real-World Rollout Visualizations · p. 14, Figure A1 · p. 15