PaperlayerView exact revision on arXiv

Source-linked AI summary

IronMan: Information-Constrained Video-Action Learning for Robot Manipulation

Yuanshuo Zhang, Wenzhe Zhao, Zixing Lei, Bin Chen, Siheng Chen

arXiv:2610.07961v1cs.RO

TL;DR

Video features preserve rich visual detail that is not naturally aligned with action generation, and policies may rely on incidental cues that impair generalization under shifts. IronMan compresses one-step video features into action-relevant world tokens; it reports strong ID and OOD results, including 79.1% on LIBERO-Plus, while maintaining efficient inference.

  • Problem

    Video-action interfaces do not explicitly suppress visual features irrelevant to action generation, and incidental cues can undermine policy generalization under visual shifts.

    Source: §1 · p. 1

  • Method

    IronMan uses a dynamics-aware bottleneck to aggregate noisy one-step video features into compact world tokens under an information constraint, with video and action supervision.

    Source: §1 · p. 2

  • Results

    IronMan achieves 99.0% on LIBERO-Long and 79.4% on RoboTwin c2c, and 79.1% on LIBERO-Plus, surpassing DiT4DiT by 10.4%.

    Source: §4.2 · p. 6, §4.3 · p. 7

  • Takeaways & Limitations

    Across simulation and real-world evaluations, IronMan demonstrates strong ID performance and OOD robustness while maintaining efficient inference.

    Source: §1 · p. 2, §6 · p. 10

Abstract

from arXiv · show

Video Action Models (VAMs) couple visual dynamics modeling with action generation for robot manipulation. However, video representations are not naturally suited to action generation, as exposing the action policy to excessive visual detail can impair its generalization ability. Therefore, we introduce IronMan (Information-constRained videO-actioN learning for robot MANipulation), a robust video-action learning framework built on the information bottleneck principle. The core principle of this framework is to impose information constraints that suppress irrelevant visual information while preserving action-relevant dynamics cues. IronMan employs a dynamics-aware bottleneck that distills noisy, entangled one-step video features into compact world representations. Extensive simulation and real-world experiments demonstrate strong in-distribution (ID) performance and out-of-distribution (OOD) robustness while maintaining efficient inference. IronMan achieves success rates of 99.0% on LIBERO and 79.4% on RoboTwin clean2clean, outperforming all the evaluated baselines. Under OOD shifts, IronMan achieves a success rate of 79.1% on LIBERO-Plus, exceeding the strongest baseline by 10.4 percentage points. Project page: https://youngsoul0731.github.io/ironman-project-page/

1 INTRODUCTION

Video features retain rich visual detail that is not aligned with action generation, and policies can rely on incidental visual correlations that fail under shifts. IronMan constrains the video-action interface to preserve action-relevant dynamics while suppressing redundancy.

1 INTRODUCTION

  • Excessive video detail is not naturally aligned with action generation, while existing projections and attention interfaces do not explicitly suppress features irrelevant to actions.The resulting sensitivity to incidental cues such as camera noise, lighting, and backgrounds can degrade generalization in unseen scenes.Source: §1 · p. 1
  • The challenge is to preserve dynamics essential for manipulation while avoiding spurious visual correlations that can undermine robustness under visual distribution shifts.The motivation figure contrasts unconstrained access to irrelevant visual signals with IronMan’s bottleneck for retaining predictive dynamics and suppressing redundancy.Source: §1 · p. 1, Figure 1 · p. 2
  • IronMan applies an information-constrained bottleneck to one-step video features, aggregating them into a compact latent distribution that forms world tokens for action generation.The framework combines this interface with video and action supervision; one-step features reduce inference overhead relative to inverse-dynamics-model-based VAMs.Source: §1 · p. 2
  • IronMan is evaluated across two simulation datasets, four ID and OOD benchmarks, and three real-world manipulation tasks with background and layout shifts.The authors report strong ID performance and OOD robustness while maintaining efficient inference.Source: §1 · p. 2

2 RELATED WORK

Related work spans VLA models, video-action models, and information-bottleneck methods. IronMan builds on their dynamics and representation-learning ideas while targeting noisy, redundant video features at the action interface.

2.1 VISION LANGUAGE ACTION MODELS

  • VLA models transfer pretrained vision-language knowledge to robot manipulation, but static image-text pretraining provides little direct supervision about temporal evolution and physical dynamics.Unlike approaches that inject dynamics into VLMs during post-training, this work uses a pretrained video generative model as its source of physical-dynamics priors.Source: §2.1 · p. 3

2.2 VIDEO ACTION MODELS

  • Explicit future-frame prediction can guide action generation but requires full video denoising, whereas implicit methods extract one-step video features that retain redundant reconstruction cues.Existing interfaces connect those features to action tokens through cross-attention or learned projections without explicitly constraining transferred information.Source: §2.2 · p. 3

2.3 INFORMATION BOTTLENECK

  • Prior information-bottleneck methods compact visuomotor representations, but the problem of isolating control-relevant information from noisy, entangled video features remains largely unexplored.Video features jointly encode rich visual detail and dynamics, making it difficult to isolate information useful for control.Source: §2.3 · p. 3

3 METHODOLOGY

IronMan turns one-step video features into compact world tokens for action generation. Its architecture aggregates spatial and temporal cues, and its joint objective balances action prediction with information compression while retaining video supervision.

3 METHODOLOGY

  • The methodology introduces an information-constrained interface that transforms one-step video features into compact world tokens for action generation.It comprises theoretical preliminaries, the architecture, and a joint training objective.Source: §3 · p. 3

3.1 PRELIMINARIES

  • Video and action prediction are jointly modeled, while the information-bottleneck objective favors representations that retain target-relevant information and discard irrelevant input information.The bottleneck balances compression against prediction; β controls that balance.Source: §3.1 · p. 4

3.2 IRONMAN ARCHITECTURE

  • A spatio-temporal aggregation module and Gaussian head compress video features into world tokens, which the action policy combines with robot-state embeddings to generate actions.The aggregation uses query tokens with spatial cross-attention to video tokens and temporal self-attention across queries; at inference, the latent is the Gaussian mean.Source: §3.2 · pp. 4–5, Figure 2 · p. 5

3.3 TRAINING OBJECTIVE

  • The training objective combines an action flow-matching loss with a KL-divergence penalty on the latent distribution and video flow-matching supervision.The action-centric term trades off action prediction against compression, while joint video supervision trains the video backbone to model visual dynamics.Source: §3.3 · p. 5

4 SIMULATION EXPERIMENTS

Across simulation benchmarks spanning in-distribution and out-of-distribution settings, IronMan reports strong performance, including 79.1% on LIBERO-Plus and 79.4% on RoboTwin clean2clean. Ablations and visualizations examine the bottleneck and attention patterns.

4.2 IN-DOMAIN EVALUATION RESULTS

  • IronMan achieved 79.4% on RoboTwin clean2clean and the highest success rate on LIBERO, including 99.0% on LIBERO-Long.The LIBERO-Long result highlights performance on long-horizon manipulation tasks.Source: Table 1 · p. 6, §4.2 · p. 6

4.1 SETUP

  • RoboTwin Clean has 50 tasks with 50 clean demonstrations each; LIBERO covers 40 tasks across four suites, evaluated under ID and OOD protocols.The evaluations also compare VLA and VAM baselines.Source: §4.1 · p. 6

4.3 OUT-OF-DISTRIBUTION EVALUATION RESULTS

  • 79.1% on LIBERO-Plus exceeded the strongest baseline, DiT4DiT, by 10.4%; IronMan also reached 12.1% on RoboTwin clean2random, ranking second.The comparison used VAM baselines without large-scale embodied pretraining and reported latency without engineering optimizations.Source: Table 2 · p. 7, §4.3 · p. 7

4.4 ABLATION STUDY

  • Combining the dynamics-aware bottleneck and information bottleneck reached 79.1% overall, versus 74.3% for DB alone and 73.2% for IB-only against the 73.4% baseline.Applying IB directly to raw video features did not improve on the baseline; the results support complementary roles for structured dynamics extraction and explicit information constraints.Source: Table 3 · p. 7, §4.4 · p. 7

4.5 BOTTLENECK MECHANISM ANALYSIS

  • All tested bottleneck configurations outperformed the 73.4% baseline; intermediate β performed best, while capacity and constraint strength traded off retained control information against irrelevant detail.A small bottleneck limits representation capacity, whereas a larger one may expose the action head to irrelevant visual details; overly strong constraints can discard information needed for control.Source: §4.5 · pp. 7–8, Figure 3 · p. 8

4.6 VISUALIZATION ANALYSIS

  • Under a LIBERO-to-LIBERO-Plus background shift, IronMan focused more selectively on gripper–object interactions, while the baseline also attended to task-irrelevant backgrounds.The comparison covers reaching and grasping stages.Source: Figure 4 · p. 8, §4.6 · p. 8

5 REAL-WORLD EXPERIMENTS

IronMan is evaluated on three real-world manipulation tasks under in-distribution, background-shift, and layout-shift settings, with results favoring IronMan over FastWAM across settings.

Setup

  • IronMan is evaluated on flower insertion, fruit arrangement, and parcel organization across AGIBOT G2 and ROBOTERA M7, with 100 demonstrations collected via teleoperation for each task.Source: §5 · p. 8

Results

  • 91.11 (ID average normalized score) and 56.67 (Background average score) put IronMan ahead of FastWAM by 23.89 and 50.28, respectively.Figure 5 summarizes real-world results under ID, Background, and Layout settings.Source: Figure 5 · p. 9, §5 · p. 9
  • 66.11 (Layout average score) is highest, exceeding π0.5 by 4.72 and FastWAM by 30.28.The authors describe the results as comparable real-world robustness despite lacking π0.5-scale embodied pretraining.Source: Figure 5 · p. 9, §5 · p. 9

Case Analysis

  • Under background shift in flower insertion, IronMan grasps the target flowers and transfers them to the vase, while FastWAM fails to establish a secure grasp.The case analysis reports more accurate grasping and placement by IronMan than FastWAM in this example.Source: §5 · p. 9, Figure 6 · p. 9

6 CONCLUSIONS

IronMan aligns visual representations with robot control through a dynamics-aware bottleneck that distills one-step video features into compact world tokens. The paper reports strong in-domain performance, out-of-distribution robustness, and efficient inference, with ablations and real-world evaluations supporting the framework.

Conclusion

  • IronMan combines spatio-temporal aggregation and information regularization to distill one-step video features into compact world tokens.The framework is described as aligning visual representations with robot control.Source: §6 · pp. 9–10
  • IronMan achieves strong in-domain performance and out-of-distribution robustness while maintaining efficient inference.Source: §6 · p. 10
  • Ablations validate complementary roles for aggregation and regularization, while real-world evaluations further support robustness.Source: §6 · p. 10

A.1.1 TRAINING AND INFERENCE ALGORITHMS

The algorithms train video generation and action generation jointly with an information constraint, then reuse extracted world tokens during action inference. The action-conditioning features come from a separate one-step forward pass over Gaussian future latents and the current observation.

Training and inference

  • A separate one-step forward pass extracts action-conditioning features from Gaussian future latents and the current observation, sharing Video DiT parameters with video training.The conditioning frame is excluded from the video loss, and the Gaussian KL is averaged over world tokens.Source: §A.1.1 · p. 14

Inference algorithm

  • At inference, the bottleneck computes world tokens once from the extracted features, and those tokens are reused throughout action denoising.The action inference algorithm returns an action chunk conditioned on the world tokens, robot state, and instruction.Source: §A.1.1 · pp. 14–15

Training algorithm

  • Training combines video flow-matching loss, action flow-matching loss, and a β-weighted Gaussian KL information constraint.The algorithm samples minibatches of observations, instructions, robot state, and action chunks, then updates the video, bottleneck, and action parameters by backpropagation.Source: §A.1.1 · p. 14

A.1.2 TRAINING CONFIGURATIONS

The training configurations first fine-tune the video backbone on simulation data, then train the video model, dynamics-aware bottleneck, and action model together. Dataset-specific camera inputs and resolutions are used for LIBERO and RoboTwin Clean.

Training configurations

  • The video backbone is first fine-tuned with video flow-matching loss, followed by joint training of the Video DiT, dynamics-aware bottleneck, and Action DiT using the equation 10 objective.Source: §A.1.2 · p. 15

LIBERO

  • LIBERO training combines external and wrist camera images resized to 224 × 224 into a 224 × 448 video frame, using the four LIBERO suites.The full configurations are referenced in Table 4a.Source: §A.1.2 · p. 15, Table 4 · p. 15

RoboTwin Clean

  • RoboTwin Clean training uses its 50-task clean-setting dataset and composes resized head, left-wrist, and right-wrist camera images into a 384 × 320 frame.The full configurations are referenced in Table 4b.Source: §A.1.2 · p. 15, Table 4 · p. 15

A.2 VISUALIZATION ANALYSIS

Across visualizations, IronMan focuses on gripper–object interactions as manipulation progresses, unlike the baseline’s broader attention; it also shows lower action deviations under visual perturbations.

A.2 VISUALIZATION ANALYSIS

  • Across four object examples, IronMan’s attention shifts from the gripper toward the gripper–object interaction and remains localized during lifting, unlike the baseline’s broad focus.The examples include different object arrangements and background shifts from LIBERO to LIBERO Plus.Source: §A.2 · pp. 15–16, Figure 7 · p. 16, Figure 8 · p. 16

A.3.1 TASK DEMONSTRATIONS

  • Three real-world tasks cover flower placement, fruit placement, and parcel shelving on two robot platforms, each tested in ID, Background, and Layout settings.The OOD settings change tabletop backgrounds or object positions and orientations; the protocol uses 10 trials per task per setting.Source: §A.3.1 · pp. 17–18

A.3.2 SCORE PROTOCOL

  • Real-world trials are scored by the highest sequential completion stage achieved, then reported as normalized task scores from 0 to 100.The same sequential criteria apply across ID, Background, and Layout settings.Source: §A.3.2 · p. 18, Table 5 · p. 18

A.4.1 INTERVENTION EXPERIMENTS

  • IronMan has lower median action L1 deviations and narrower interquartile ranges than the baseline across four visual perturbation categories.Reductions are particularly pronounced for camera viewpoints, lighting, and sensor noise; background-texture perturbations show a smaller median reduction.Source: Figure 12 · p. 20, §A.4.1 · p. 20

Related papers

Loading 2610.07961v1…