PaperlayerView exact revision on arXiv

Source-linked AI summary

DynaFLIP: Rethinking Robotics Perception via Tri-Modal-Dynamics Guided Representation

Jusuk Lee, Seungjae Lee, Jonghun Shin, Hoseong Jung, Sungha Kim, Daesol Cho, H. Jin Kim, Jia-Bin Huang, Furong Huang

arXiv:2605.30350v1cs.ROcs.LG

TL;DR

Robot learning often relies on static or vision-language encoders while leaving motion understanding to downstream control. DynaFLIP uses image transitions, language, and 3D flow to shape an image-only encoder, consistently outperforming baselines across simulated and real-world manipulation, with gains up to 22.5% under real-world OOD perturbations.

  • Problem

    Existing robot-learning encoders emphasize static recognition or vision-language alignment, providing limited dynamics awareness for manipulation across diverse environments and tasks.

  • Method

    DynaFLIP aligns image transitions, language, and 3D flow through simplex-volume minimization with cosine regularization and contrastive learning to train an image-only encoder.

  • Results

    Up to 22.5% improvement over the strongest baseline under real-world OOD perturbations accompanies consistent gains across simulated and real-world manipulation policies.

  • Takeaways & Limitations

    Training visual representations to encode how scenes change under action can improve robot generalization across downstream manipulation settings.

  • Takeaways & Limitations

    Task-irrelevant motion in pre-training videos can inject noisy supervision because 3D flow captures all motion after camera-motion compensation.

Abstract

from arXiv · show

Robot manipulation critically depends on perception that preserves the action-relevant aspects of a scene. Yet most robot learning pipelines are built upon visual encoders pre-trained for static recognition or vision-language alignment, leaving motion understanding to downstream policies. We introduce DynaFLIP, a dynamics-aware multimodal pre-training framework that pushes motion understanding upstream into perception. We construct image-language-3D flow triplets from heterogeneous human and robot videos, and use these triplets as training-time supervision to shape an image-only encoder. Our key idea is to encourage the three modalities to span a small simplex volume in the shared hyperspherical space -- a smaller simplex volume indicating stronger alignment. To avoid the geometric ambiguity and trivial collapse of naive volume minimization, we combine simplex-volume minimization with a cosine regularizer and a contrastive objective. Our analyses show that DynaFLIP focuses on control-relevant regions critical for manipulation. The resulting dynamics-aware representations serve as reusable visual backbones and consistently outperform baselines across diverse downstream policies, including VLAs. We validate this across diverse simulation and real-world setups, with gains reaching +22.5% under out-of-distribution scenarios. Our results suggest that robot generalization improves when visual representations are trained to encode not just what is present, but how the world changes under action.

1 Introduction

DynaFLIP treats robot generalization as a perception problem: visual representations should encode dynamics- and control-relevant structure, not merely static visual content. It transfers image-transition, language, and 3D-flow supervision into an image-only encoder through joint simplex-based multimodal alignment.

  • Motivation: Robot learning systems commonly reuse CLIP, SigLIP, and DINOv2 encoders across imitation-learning and Vision-Language-Action policies.This practice inherits the assumption that perception can be borrowed from pretrained visual encoders.
  • Motivation: Single images may not reveal which scene aspects are causally relevant for action, motivating other modalities as training-time supervision rather than test-time inputs.Language and realized state changes can provide complementary evidence for shaping the visual representation.
  • DynaFLIP: The framework jointly aligns all modality embeddings through their shared simplex, addressing the limitations of anchor-based objectives and avoiding geometric ambiguity and trivial collapse.The contribution is framed as higher-order multimodal alignment that distills dynamics- and control-relevant structure into visual representations.
  • DynaFLIP: DynaFLIP uses image transitions, language, and 3D flow to shape an image-only encoder and produce control-relevant visual representations for downstream manipulation.Its alignment objective minimizes the simplex volume spanned by the three modalities in a shared embedding space.

2 Method

DynaFLIP aligns image transitions, language, and 3D flow in a shared hyperspherical space using simplex geometry, cosine regularization, and contrastive learning. Temporal contrastive and actor objectives further encode longer-horizon structure and manipulation dynamics, using signals derived from RGB videos.

  • Tri-modal alignment: DynaFLIP maps image transitions, language, and 3D flow to ℓ2-normalized embeddings, representing visual change, semantic intent, and physical motion.The modalities are denoted zI, zL, and zF on the unit sphere.
  • Tri-modal alignment: Simplex-volume minimization captures mutual three-way alignment beyond anchor-based pairwise contrastive learning.In the three-modal case, the simplex volume reduces to the triangle area spanned by the embeddings.
  • Optimization safeguards: A language–flow cosine regularizer resolves geometric ambiguity by penalizing flat, near-zero-area configurations with poor mutual alignment.The energy combines triangle-area minimization with −α⟨zL, zF⟩, where α ≥ 0 balances the two terms.
  • Optimization safeguards: An InfoNCE-style objective with mismatched modality tuples prevents trivial collapse by forcing matched tuples to achieve lower energy than negatives.Negatives are formed by mismatching one or more modality embeddings across samples in the batch.
  • Auxiliary objectives: Temporal contrastive and actor losses extend alignment across trajectories and encourage single-frame image features to predict 3D motion explicitly.Nearby frames are pulled closer than distant frames, while the actor objective minimizes mean squared error between predicted and ground-truth flow.

3 Experiments

Experiments across simulation and real-world manipulation show that DynaFLIP preserves control-relevant information, improves downstream success across policies and settings, and benefits from its tri-modal alignment and auxiliary objectives.

  • Benchmarks: DynaFLIP is evaluated on three simulation benchmarks and three real-world manipulation tasks spanning MetaWorld, RLBench, LIBERO, and physical robot setups.MetaWorld includes 15 tasks with 25 demonstrations per task; LIBERO includes LIBERO-90, LIBERO-Goal, LIBERO-Object, LIBERO-Spatial, and LIBERO-Long.
  • Qualitative Analysis: DynaFLIP attends to manipulated objects and interaction regions while producing more spatially coherent, object-level feature structures than baselines.Grad-CAM highlights action-influential regions, and PCA reveals the organization of patch features.
  • Control-Relevant Representations: With image encoders frozen and only a lightweight three-layer MLP trained, DynaFLIP achieves the highest downstream success rate with high control-relevant scores on MetaWorld and RLBench.The control-relevant score probes information about robot joint angles, end-effector pose, and task-relevant object pose and shape.
  • Downstream Policy Performance: DynaFLIP achieves the highest mean LIBERO success rate in both frozen and fine-tuned settings, outperforming all baselines and remaining effective without encoder adaptation.The comparison uses diffusion policies, with Frozen and LoRA Fine-tuned settings evaluated by success rate (%).
  • Ablations: Removing 3D flow, language, simplex-guided alignment, contrastive learning, cosine regularization, or auxiliary losses degrades performance, showing that each design choice contributes to DynaFLIP.The largest ablation drop occurs when the contrastive framework is removed, while removing Ltcn causes a larger decline than removing the other auxiliary loss.

4 Related work

Prior robotic visual representation work centers on self-supervised or contrastive vision-language pre-training, but these paradigms primarily learn from static data. DynaFLIP addresses the resulting dynamics gap by jointly aligning image transitions, language, and 3D flow to emphasize control-relevant regions.

  • Visual Representations for Robotic Manipulation: Robotic visual pre-training mainly follows self-supervised visual or contrastive vision-language paradigms.Self-supervised models capture global context and local detail, while lacking a direct language interface limits open-vocabulary and instruction-following use.
  • Visual Representations for Robotic Manipulation: Both major paradigms primarily learn from static data, leaving them unaware of dynamics that determine how scenes change during manipulation.This matters because manipulation success depends on interaction-driven scene changes, not only object and instruction recognition.
  • Visual Representations for Robotic Manipulation: DynaFLIP aligns image transitions, language, and 3D flow so its encoder can focus on control-relevant rather than merely visually salient regions.These three signals provide transition-centric supervision for dynamics-aware perception.
  • Visual Representations for Robotic Manipulation: Existing robotic representation objectives span single-modality self-supervision and multimodal alignment with language, action, or robot proprioception, but do not jointly align all three modalities.DynaFLIP instead uses a simplex-based formulation to enable mutual alignment among image transitions, language, and 3D flow.

5 Conclusion · Appendix · A Additional Related Works

DynaFLIP moves motion understanding into perception by jointly aligning image transitions, language, and 3D flow, producing reusable visual representations focused on control-relevant regions. Its main limitations are modest pre-training scale and potentially noisy supervision from task-irrelevant motion in uniformly sampled 3D flow.

  • 5 Conclusion: DynaFLIP is a dynamics-aware 3D flow-language-image pre-training framework that moves motion understanding upstream into perception.It jointly aligns image transitions, language, and 3D flow through a simplex-based formulation.
  • 5 Conclusion: The framework combines simplex-based alignment with cosine regularization and contrastive learning to address optimization pitfalls.These components support the joint alignment of image transitions, language, and 3D flow.
  • 5 Conclusion: DynaFLIP learns visual representations that focus on control-relevant regions for manipulation.The representations are shaped by jointly aligning image transitions, language, and 3D flow.
  • 5 Conclusion: DynaFLIP transfers strongly as a reusable visual representation across simulated and real-world manipulation.The supplied conclusion passage states this transfer result but is truncated before providing further quantitative detail.
  • 5 Conclusion: 260K trajectories are used for DynaFLIP pre-training, fewer than the data scales used by several large-scale visual and vision-language baselines.The authors identify this smaller pre-training scale as a limitation and motivate scaling to larger human and robot video corpora.
  • 5 Conclusion: A uniform 20 × 20 grid of keypoints captures all scene motion after camera-motion compensation, including task-irrelevant motion.Videos containing such motion may inject noisy supervision, motivating keypoint sampling focused on the agent and task-relevant objects.

A.1 Pre-training Objectives for Robotic Representations

Robotic representation pre-training has progressed from static-image self-supervision toward objectives tailored to robotics. Early methods use MAE, implicit value functions, or video-derived affordances to learn visual features and task-relevant representations.

  • MVP and VC-1 apply Masked Autoencoder objectives to large-scale human datasets for visual feature learning.
  • VIP learns implicit value functions that encode distance-to-goal representations.
  • HRP extracts human affordances from videos as a robotic representation pre-training signal.

B Mathematical Proofs and Theoretical Details … C Dataset Construction

DynaFLIP’s theory shows why simplex-volume alignment alone is insufficient and explains how cosine regularization restores reliable pairwise alignment. The dataset-construction section outlines the heterogeneous video sources and conversion of raw videos into image–language–3D flow triplets.

  • B.1 Generalized Simplex Volume: For m modalities, the generalized simplex volume measures the shared latent configuration, reducing to triangle area for DynaFLIP’s three modalities.Smaller volume indicates stronger joint multimodal alignment.
  • B.2 Contrastive Learning with Simplex-Guided Energy: The simplex-guided energy preserves an energy-based contrastive-learning structure while modifying the positive alignment energy with triangle-area and cosine terms.The alignment gradient is a linear combination capturing higher-order geometry and directional consistency.
  • B Mathematical Proofs and Theoretical Details: The theoretical analysis identifies geometric ambiguity and conflicting alignment gradients as the two central limitations of simplex-volume-only optimization.These pitfalls motivate combining simplex-volume minimization with cosine regularization and contrastive learning.
  • B.3 Why Simplex-Volume Alone is Insufficient: Volume-only alignment is insufficient because low-volume configurations can be ambiguous and their induced gradients can conflict.The three-modal analysis characterizes both limitations directly.
  • B.3.1 Ambiguity of Low-Volume Configurations: Low simplex volume does not guarantee pairwise alignment because embeddings may collapse partially or become nearly collinear.In the example, A(x, y, z) = | sin θ| →0 while ⟨x, y⟩ = −1 and ∥x−y∥ = 2.
  • B.3.2 Conflicting Alignment Directions in Volume-Induced Gradients: Volume-induced gradients sum multiple edge-wise pulls rather than following one direction, so differently oriented pulls can partially cancel and weaken updates.Aligned pulls reinforce each other, whereas divergent pulls produce a weaker effective signal.
  • B.4 Mitigating Volume-Only Pitfalls with Cosine Regularization: Cosine regularization adds explicit pairwise constraints that penalize distant modalities, discouraging degenerate low-volume configurations and preserving a non-vanishing alignment signal.The cosine term directly increases similarity for selected modality pairs even when volume-induced pulls cancel.
  • C Dataset Construction: The dataset-construction section describes heterogeneous human and robot video sources and a pipeline that converts raw videos into image–language–3D flow triplets.These triplets provide the multimodal data used by DynaFLIP.

C.1 Dataset Composition

The pre-training dataset combines heterogeneous human and robot videos into 260K image–language–3D flow triplets, covering varied objects, environments, viewpoints, and manipulation styles.

  • Dataset composition: The dataset draws from heterogeneous human and robot video sources to span diverse objects, environments, camera viewpoints, and manipulation styles.This diversity is the stated motivation for combining the two video domains.
  • Dataset composition: 260K image–language–3D flow triplets comprise the dataset, including 190K robot trajectories and 70K human trajectories.The robot portion includes AgiBot (135K), Droid (20K), Open X-Embodiment (17K), and BridgeData V2 (18K).

C.2 Dataset Generation Pipeline · D Experiment Details

The dataset-generation pipeline converts raw videos into aligned image–language–3D flow triplets through frame sampling and three parallel processing branches. It generates manipulation-intent instructions and camera-compensated, screen-aligned 3D motion representations from tracked points, depth, and camera pose.

  • C.2 Dataset Generation Pipeline: The pipeline converts raw videos into aligned image–language–3D flow triplets by directly sampling frames and omitting event chunking and speed retargeting.Frame sampling approximately matches effective temporal resolution across datasets with different frame rates.
  • C.2 Dataset Generation Pipeline: For each sampled trajectory, a VLM generates imperative, detailed, and multi-step language instructions describing manipulation intent.The VLM uses representative trajectory frames and a prompt that decomposes tasks into sequential subgoals.
  • C.2 Dataset Generation Pipeline: A uniform 20 × 20 grid of keypoints is placed on an early reference frame and tracked throughout each trajectory.The early reference frame is selected because the first frame may not contain the robot or human demonstrator.
  • C.2 Dataset Generation Pipeline: Tracked points are represented in 3D as (x, y, z), preserving spatial alignment with the original image while retaining physically meaningful motion.This representation connects image-aligned observations with motion expressed in 3D space.
  • C.2 Dataset Generation Pipeline: The pipeline uses SpatialTrackerV2, CoTracker3, and camera geometry to estimate depth, pose, and tracked point trajectories for 3D flow construction.The figure identifies three parallel branches for language generation, camera-pose/depth estimation, and 2D point tracking.
  • C.2 Dataset Generation Pipeline: To compensate for camera motion, reconstructed 3D points are transformed into the reference camera coordinate system and projected back using camera intrinsics.The resulting flow is stored as a screen-aligned sequence with depth values defined in the reference camera frame.
  • C.2 Dataset Generation Pipeline: The final formulation isolates true scene motion rather than conflating it with viewpoint-dependent image-plane displacement.Camera extrinsics perform the world-to-reference transformation before screen-aligned storage.

D.1 Pre-training DynaFLIP · D.2 MetaWorld and RLBench · D.3 LIBERO

DynaFLIP pretraining combines trainable image, adapted language, and temporal 3D-flow encoders to produce dynamics-aware representations. Downstream evaluations test these frozen or adapted representations across MetaWorld, RLBench, and LIBERO using lightweight policies and diffusion policy.

  • D.1 Pre-training DynaFLIP: DynaFLIP uses trainable DINOv2-Base image features, a frozen T5-Base encoder with a learnable adapter, and a temporal 3D-flow encoder.The image backbone is ViT-B/14 with 768-dimensional tokens; language instructions are limited to 77 tokens, while flow inputs contain K timesteps of 20 × 20 × 3 data.
  • D.1 Pre-training DynaFLIP: The image encoder forms per-frame embeddings from [CLS] and average-pooled patch tokens, then fuses adjacent sampled frames into an image-transition embedding.The same procedure is applied to every sampled frame in a clip before transition fusion.
  • D.1 Pre-training DynaFLIP: Pretraining samples five ordered frames per video, using initial and final frames from the first and last 10% and three intermediate frames.The ordered sequence supports construction of sequential transition pairs rather than a single transition.
  • D.2 MetaWorld and RLBench: MetaWorld and RLBench evaluate frozen visual representations with only a three-layer MLP policy trained from third-person RGB observations and proprioceptive state.MetaWorld covers 15 Sawyer-arm tasks across difficulty levels, while RLBench covers six Franka Panda tasks with 100 demonstrations per task.
  • D.2 MetaWorld and RLBench: Both MetaWorld and RLBench methods train for 100 epochs, evaluate every 10 epochs using 25 rollouts, and report the best checkpoint’s average rollout success rate.RLBench uses only the front-view camera as visual input, and its task-wise success rates are reported separately.
  • D.3 LIBERO: DynaFLIP is evaluated on LIBERO-90, LIBERO-Goal, LIBERO-Object, LIBERO-Spatial, and LIBERO-Long, with 90 tasks in LIBERO-90 and 10 in each other suite.Each non-LIBERO-90 suite provides 50 demonstrations per task.
  • D.3 LIBERO: LIBERO uses Diffusion Policy with a U-Net backbone and third-person RGB conditioning, while the primary setting freezes image and language encoders.The policy uses DDIM action generation with 100 forward diffusion steps, 10 denoising steps, prediction horizon 32, execution horizon 16, and observation history 1.
  • D.3 LIBERO: Each LIBERO suite trains a separate diffusion policy for 200 epochs, evaluates every 20 epochs with 20 rollouts per task, and reports the best checkpoint’s average success rate.An additional LoRA setting jointly adapts both encoders with the diffusion policy.

D.4 Real-world Robot · D.5 Control-Relevant Metric · E Additional Experimental Results

DynaFLIP is evaluated on real-world manipulation with parameter-efficient π0.5 adaptation and a simulator-grounded metric for control-relevant representation quality. The supplied passages describe three tasks, closed-loop evaluation settings, and probe-based scoring, but provide no additional results for Section E.

  • D.4 Real-world Robot: The real-robot platform is a fixed-base UR3 with a two-finger gripper, two 224 × 224 RGB cameras, and a 7D proprioceptive state.The cameras are third-person and wrist-mounted views; the proprioceptive state contains end-effector pose and gripper state.
  • E Additional Experimental Results: The supplied passages contain no substantive passage for E Additional Experimental Results.The provided evidence covers D.4 Real-world Robot and D.5 Control-Relevant Metric only.
  • D.4 Real-world Robot: DynaFLIP is evaluated on Pick <object> into Sink, Pour almonds into <object>, and Unfold Towel, covering rigid and deformable manipulation.The tasks use 90, 80, and 50 demonstrations, respectively.
  • D.4 Real-world Robot: The frozen π0.5 backbone and visual encoder receive auxiliary features through trainable projection, action-expert, and per-layer injection modules.Zero initialization makes the initial policy equivalent to the pre-trained VLA while the injected visual signal activates during training.
  • D.4 Real-world Robot: Policies are evaluated through closed-loop rollouts, with 20 rollouts per setting and OOD visual-spatial and semantic perturbations for the first two tasks.Unfold Towel is evaluated only in-distribution, and success requires completing the instructed task within the episode horizon.
  • D.5 Control-Relevant Metric: The control-relevant metric trains lightweight probes to predict simulator state from frozen visual features, using RoI-pooled maps for objects and global features for scene state.Simulator state combines object-level information with robot joint configuration and end-effector pose.
  • D.5 Control-Relevant Metric: The score S_m min-max normalizes negative mean-squared-error prediction scores across models for each state dimension and averages them; larger S_m indicates more preserved control information.The metric provides a unified scale for comparing representations.

E.1 LIBERO Results with Paired Image Encoders · E.2 Grad-CAM visualizations · E.3 PCA visualizations

On LIBERO, DynaFLIP improves paired frozen image-encoder performance, while visualizations show representations that emphasize control-relevant regions and provide PCA views of learned features.

  • E.1 LIBERO Results with Paired Image Encoders: The paired-encoder LIBERO evaluation trains only the diffusion policy and reports success rate (%).DINOv2 supplies fine-grained spatial features, whereas language-aligned encoders provide high-level semantics.
  • E.1 LIBERO Results with Paired Image Encoders: The LIBERO comparison pairs DINOv2 with CLIP, SigLIP, or DynaFLIP under the same frozen-encoder configuration.Features are concatenated before being passed to the diffusion policy, while the corresponding language encoder is paired with each setup.
  • E.1 LIBERO Results with Paired Image Encoders: DINOv2 + DynaFLIP achieves the highest mean success rate on LIBERO, outperforming DINOv2 + CLIP and DINOv2 + SigLIP.DynaFLIP combines language alignment for instruction following with dynamics-aware focus on control-relevant manipulation regions.
  • E.2 Grad-CAM visualizations: Grad-CAM visualizations identify image regions contributing most to downstream action prediction for each frozen image encoder.The scalar target is the negative mean squared error between the trained three-layer MLP policy head’s predicted action and the ground-truth action.
  • E.2 Grad-CAM visualizations: Additional Grad-CAM examples show DynaFLIP consistently focusing on task-relevant objects and interaction regions.Baseline encoders more often exhibit diffuse attention or emphasize less control-relevant regions.
  • E.3 PCA visualizations: PCA visualizations project each encoder’s spatial features to three principal components and map them to RGB.ViT-based encoders use patch tokens, while CNN-based encoders use the 7 × 7 output of the final convolution block.
  • E.3 PCA visualizations: Additional PCA visualizations complement the paper’s qualitative analysis of learned representations.The visualizations are presented as supplementary views of encoder features across the evaluated models.
Loading 2605.30350v1…