PaperlayerView exact revision on arXiv

Source-linked AI summary

FRAPPE: Infusing World Modeling into Generalist Policies via Multiple Future Representation Alignment

Han Zhao, Jingbo Wang, Wenxuan Song, Shuai Chen, Yang Liu, Yan Wang, Haoang Li, Donglin Wang

arXiv:2602.17259v1cs.RO

TL;DR

Existing world-modeling policies can overfit redundant pixels, accumulate inference errors, and inherit biases from single visual representations. FRAPPE aligns future representations with multiple visual foundation models through two-stage progressive parallel expansion, and experiments report stronger performance, efficiency, and data scaling in simulation and real-world tasks.

  • Problem

    Existing approaches face redundant pixel reconstruction, inference error accumulation, and inductive biases from representations learned from a single visual task.

  • Method

    FRAPPE uses mid-training future-representation alignment followed by parallel post-training that aligns multiple visual representations while expanding computational capacity.

  • Results

    FRAPPE demonstrates advantages in data efficiency, parameter efficiency, and overall performance on RoboTwin and real-world evaluations, surpassing baseline models.

  • Takeaways & Limitations

    FRAPPE can leverage large-scale egocentric human-operation data to enhance generalization beyond fine-tuning on target domains.

Abstract

from arXiv · show

Enabling VLA models to predict environmental dynamics, known as world modeling, has been recognized as essential for improving robotic reasoning and generalization. However, current approaches face two main issues: 1. The training objective forces models to over-emphasize pixel-level reconstruction, which constrains semantic learning and generalization 2. Reliance on predicted future observations during inference often leads to error accumulation. To address these challenges, we introduce Future Representation Alignment via Parallel Progressive Expansion (FRAPPE). Our method adopts a two-stage fine-tuning strategy: In the mid-training phase, the model learns to predict the latent representations of future observations; In the post-training phase, we expand the computational workload in parallel and align the representation simultaneously with multiple different visual foundation models. By significantly improving fine-tuning efficiency and reducing dependence on action-annotated data, FRAPPE provides a scalable and data-efficient pathway to enhance world-awareness in generalist robotic policies. Experiments on the RoboTwin benchmark and real-world tasks demonstrate that FRAPPE outperforms state-of-the-art approaches and shows strong generalization in long-horizon and unseen scenarios.

1 Introduction

FRAPPE addresses limitations of pixel-level world modeling and error accumulation by aligning future representations through progressive parallel expansion. Its two-stage strategy supports efficient training, human-video scaling, and improved performance across simulated and real-world settings.

  • Pixel-level reconstruction can waste computation on redundant pixels, while explicit future observations during inference may cause error accumulation.
  • The method progressively expands computation through a two-stage strategy, using mid-training alignment before parallel post-training and inference.
  • FRAPPE builds an implicit world model by aligning policy features with multiple visual latent representations, avoiding biases from single-representation learning.
  • FRAPPE can train on human video demonstrations without action annotations, lowering dependence on action-labeled data.
  • 10-15% improvement in overall performance is reported over a teleoperation-only baseline when teleoperation data is extremely limited.

2 Related Work

Prior work unifies video and action modeling, extracts predictive visual representations, and learns from human egocentric videos. FRAPPE instead uses future observations as an explicit grounding signal for dynamic modeling and policy learning.

  • Diffusion-based approaches model multimodal action distributions, while recent methods unify video generation and action prediction through shared or joint diffusion processes.
  • Video-diffusion encoders and representation-alignment methods provide predictive visual representations for downstream policy conditioning without explicit future-frame generation.
  • Robot learning from human egocentric videos commonly extracts affordances, visual trajectories, or poses to translate human motion into robotic behavior.
  • FRAPPE uses future observations as an explicit grounding signal, supporting dynamic modeling for policy learning.

3 Method

FRAPPE builds implicit world modeling into a robotic diffusion policy by aligning future latent representations with multiple visual foundation models. Its method combines future-prefix prediction, parallel prefix-and-LoRA experts, routed action aggregation, load balancing, and staged training.

  • 3.1 Preliminaries: Robotic Diffusion Transformer: RDT predicts action chunks from noisy actions, proprioception, visual tokens, and language tokens using a diffusion-transformer backbone.The backbone projects its output into the physical action space through an MLP decoder.
  • 3.2 Future Representation Alignment via Parallel Progressive Expansion: FRAPPE adds learnable future prefixes so the model predicts future visual representations from current state and action-conditioning inputs.A pretrained visual foundation model supplies future latent targets, and cosine alignment is combined with action loss.
  • 3.2 Future Representation Alignment via Parallel Progressive Expansion: During parallel scaling, separate prefix-and-LoRA experts align with dedicated teacher encoders while sharing one RDT backbone.Only prefixes and LoRA modules are trainable to reduce fine-tuning memory, with three teacher encoders used in the reported setup.
  • 3.2 Future Representation Alignment via Parallel Progressive Expansion: A lightweight router combines expert latent action representations into the final executable action chunk through a shared action head.The router produces gating weights over the parallel experts before the weighted representations are decoded.
  • 3.2 Future Representation Alignment via Parallel Progressive Expansion: Load-balancing loss prevents one expert stream from dominating training and encourages similar router-logit magnitudes across experts.The method addresses observed mode collapse, where a dominant stream can prevent other experts from updating.
  • 3.2 Future Representation Alignment via Parallel Progressive Expansion: Label smoothing assigns every expert a minimum non-zero gating weight, ensuring all experts receive updates during early training.The smoothing hyperparameter is set to 0.1 in this paper.
  • 3.2 Future Representation Alignment via Parallel Progressive Expansion: FRAPPE uses mid-training single-stream alignment with a teacher distilled from multiple visual foundation models before applying parallel training during post-training.The mid-training stage full-parameter-finetunes the model for the world-modeling objective, while the distilled encoder supports consistency between stages.

4 Simulation Experiments

FRAPPE is evaluated on RoboTwin across state-of-the-art comparisons, training paradigms, inference efficiency, and smaller policy scales. The experiments report improvements over RDT, efficient LoRA-based post-training, and strong performance on harder and smaller-scale settings.

  • Comparison with State-of-the-Arts: FRAPPE achieves the highest average success rates across all tasks in RoboTwin’s Easy setting.
  • Comparison with State-of-the-Arts: Under the Hard setting, FRAPPE surpasses π0.5 despite distractors, varied textures, lighting, and table heights challenging visual generalization.
  • Comparison of Training Paradigm: A 4.6% average success-rate improvement comes from single-stage mid-training over action-only fine-tuning under the same 20k-step budget.
  • Comparison of Training Paradigm: Post-training with LoRA improves task success and adapts to teacher representation-space changes without requiring full-parameter fine-tuning.
  • Inference Efficiency: With five denoising steps, FRAPPE increases action-generation latency by about 20 ms and uses 8.0 GB memory; with three steps, it has lower latency than baseline and higher performance.
  • Verification Experiments on Smaller-Scale Policy Model: At 130M parameters, FRAPPE improves over baseline across easy and hard settings, while LoRA post-training is only 2–3% below full-parameter fine-tuning in average success rate.

5 Real-world Experiments

Real-world experiments evaluate FRAPPE across seen and unseen variations, long-horizon manipulation, and training with action-free human egocentric data. FRAPPE shows strong generalization, achieves a 20% success rate on a challenging long-horizon task, and improves performance when combined with egocentric data.

  • Generalization: FRAPPE achieves strong performance across seen and unseen real-world variations involving lighting, height, pose, and object changes.Unseen settings represent task configurations not encountered during training.
  • Long-horizon performance: 20% success rate is achieved by FRAPPE on a challenging long-horizon task, while vanilla RDT fails in every trial.The task contains three temporally dependent subtasks and requires dual-arm coordination and fine-grained manipulation.
  • Egocentric data: FRAPPE can use action-free human egocentric videos to improve generalized world-modeling capabilities beyond robot-data-only fine-tuning.The experiments compare robot, task-specific egocentric, and web-scale egocentric data configurations.
  • Egocentric data: Web-scale egocentric data consistently improves performance on five easy-to-grasp objects and provides an inductive prior for novel objects.The downstream tasks are unrelated to the open-sourced egocentric dataset.
  • Egocentric data: Co-training substantially improves performance on five hard-to-grasp objects compared with using only a small amount of robot data.The authors associate this gain with diverse complex object geometries in the large-scale dataset.
  • Egocentric data: Combining data from different levels of the data pyramid further improves performance, motivating hierarchical integration across training stages.The framework includes large-scale action-free human egocentric data and task-specific human egocentric data.

6 Conclusion

FRAPPE combines future-representation alignment during mid-training with parameter-efficient parallel scaling during post-training. Simulation and real-world evaluations report advantages in data efficiency, parameter efficiency, overall performance, and use of large-scale egocentric human-operation data.

  • FRAPPE uses mid-training for world-modeling-enhanced visual representation alignment and parameter-efficient post-training for parallel scaling.
  • FRAPPE shows advantages in data efficiency, parameter efficiency, and overall performance across RoboTwin simulation and real-world evaluations.The comparison is against baseline models trained through direct fine-tuning.

Appendix

The appendix documents implementation details and hyperparameter analyses for alignment loss, future supervision depth, prediction horizon, and human egocentric co-training.

  • The appendix covers hyperparameter studies and the human egocentric co-training phase, including dataset characteristics, computational requirements, and data-selection rationale.

A Hyperparameter Settings

Hyperparameter studies identify settings for alignment-loss weighting, future-alignment depth, and prediction horizon used in the remaining experiments.

  • Alignment loss: λ1 = 0.05 gives the best performance as the alignment-loss coefficient.Excessive alignment-loss weight may interfere with the primary action-prediction task.
  • Future alignment depth: The 21st DiT layer gives the best performance for learnable prefix alignment in the 28-layer RDT-1B backbone.This corresponds to approximately three-quarters of the model depth.
  • Future horizon: The model achieves optimal performance when the future horizon h is set to 8.
  • The identified optimal hyperparameters are used as the standard configuration for other experiments.

B Human Egocentric Co-Training Details

The human egocentric co-training stage uses TASTE-Rob, a large-scale hand-object interaction video dataset, with one epoch of training on 8 H100 GPUs. Its fixed egocentric viewpoint is intended to support transfer to downstream robot action prediction.

  • TASTE-Rob contains 100,856 video sequences and roughly 9 million frames with high-fidelity linguistic alignment.
  • Training uses one epoch and requires 96 hours of computation on 8 H100 GPUs.
  • The dataset’s fixed egocentric viewpoint closely mirrors camera settings typically used in mainstream VLA models.
  • This camera consistency significantly enhances transferability of learned features for downstream robot action prediction tasks.
Loading 2602.17259v1…