PaperlayerView exact revision on arXiv

Source-linked AI summary

AffordanceWAM: Affordance-Aware Joint World-Action Modeling for Robot Manipulation

Jiadi You, Qize Yu, Yue Chen, Minghong Cai, Zhide Zhong, Yuran Wang, Bowen Ping, Jiaqi Liang, Zhenhao Shen, Haodong Yan, Yinchuan Li, Ruihai Wu, Xiaojuan Qi, Yingcong Chen

arXiv:2609.22332v3cs.ROcs.AIcs.CV

TL;DR

Robot manipulation needs to transfer across human and robot data, but human videos lack robot actions and existing World representations often leave interaction structure implicit. AffordanceWAM jointly predicts future visual and affordance states with robot actions, and controlled results show larger gains from human videos when affordance is included.

  • Problem

    Human videos offer diverse interactions but lack robot actions and differ in embodiment, while many WAMs leave task-relevant interaction structure implicit in visual representations.

    Source: §1 · p. 1

  • Method

    AffordanceWAM jointly predicts future RGB, Scalar Affordance, and Affordance Heatmap with continuous robot actions, using human videos for World supervision and robot trajectories for additional action supervision.

    Source: §1 · p. 2

  • Results

    With human and robot videos, affordance raises RoboCasa success from 55.1% to 69.4% and CALVIN average sequence length from 3.68 to 4.22.

    Source: §1 · p. 2, §4.3 · p. 9

  • Takeaways & Limitations

    Controlled ablations suggest human-video benefits for robot control are pronounced when affordance serves as the shared predictive representation across embodiments.

    Source: §1 · p. 2, §5 · p. 12

Abstract

from arXiv · show

Generalizable robot manipulation requires predicting how a scene will evolve, identifying where interactions are feasible, and determining how to act. Action-labeled robot videos directly supervise control but are costly and limited in diversity, whereas egocentric human videos capture diverse interactions but lack robot actions and differ in embodiment and appearance. We introduce AffordanceWAM, an affordance-aware generative World Action Model that represents object-centric spatiotemporal affordance through Scalar Affordance and Affordance Heatmap, within the generated future World. This representation grounds visual prediction in task-relevantant objects and interaction regions for action generation, and provides shared interaction targets across human and robot videos. Built on a pretrained video diffusion Transformer, AffordanceWAM uses separately parameterized World and Action Experts, coupled through Masked Joint Self-Attention, to jointly predict future RGB observations, Scalar Affordance fields, Affordance Heatmaps, and continuous robot actions under a unified flow-matching objective. Human videos supervise all three future-World streams, whereas robot trajectories additionally provide action supervision, enabling transfer without human action labels or retargeting. Experiments on RoboCasa, CALVIN ABC$\rightarrow$D, and real-world manipulation demonstrate consistent gains over RGB-only and robot-data-only baselines. Under fixed robot supervision, RoboCasa performance improves monotonically as affordance-annotated human video scales. These results support affordance as an effective interface for both vision-language-action learning and human-to-robot transfer.

1 Introduction

AffordanceWAM makes task-relevant interaction structure explicit in the predicted future World, aiming to bridge human and robot data without aligning human motions to robot actions. Experiments and controlled ablations report stronger results when affordance provides the shared representation.

1 Introduction

  • Most WAMs predict RGB frames or visual latents, leaving task-relevant objects and feasible interaction regions implicit.The paper frames the central question as which World representation supports transfer across embodiments during WAM co-training.Source: §1 · p. 1
  • AffordanceWAM forecasts Scalar Affordance and Affordance Heatmap alongside future RGB, with human clips training the World Expert and robot trajectories additionally training the Action Expert.Masked Joint Self-Attention lets action generation use RGB and Scalar Affordance features while blocking action-to-World information and Heatmap input to the Action Expert.Source: §1 · p. 2
  • With affordance, human-plus-robot training raises RoboCasa success from 55.1% to 69.4% and CALVIN average sequence length from 3.68 to 4.22.The paper also reports that AffordanceWAM outperforms strong VLAs and WAMs across simulation and real-world experiments.Source: §1 · p. 2
  • Affordance is presented as a shared interaction representation that separates shared interaction dynamics from embodiment-specific control.The paper identifies this separation as the basis for using human interaction data without human action labels or retargeting.Source: §1 · p. 2

2 Related Work

Prior World Action Models jointly predict future observations and actions, but typically leave manipulation-relevant interaction structure implicit in visual representations. AffordanceWAM instead makes that structure a shared, temporally evolving prediction target across human and robot videos.

2.1 World Action Models

  • World Action Models jointly learn future observations and actions, but their RGB or visual-latent Worlds typically leave task-relevant objects and interaction regions implicit.Earlier video-based approaches may connect visual prediction to control through separate policies, while WAMs model future observations and actions within a unified framework [32].Source: §2.1 · p. 2

2.2 Robot Learning from Human Videos

  • Human-video transfer has ranged from visual pretraining and extracted contacts or motion to retargeted actions, with recent approaches still relying on motion recovery and cross-embodiment alignment [51].AffordanceWAM offers a complementary future-state route: its Scalar Affordance and Affordance Heatmap identify task-relevant interaction regions and associate their evolution with visual context.Source: §2.2 · p. 3

2.3 Affordance Learning for Robotic Manipulation

  • Unlike approaches that use affordances as policy inputs or intermediate predictions [19] [45], AffordanceWAM treats evolving Scalar Affordance and Affordance Heatmap as shared future-World targets.Human and robot videos share these prediction targets, while only robot trajectories supervise actions; the Action Expert reads RGB and Scalar Affordance features.Source: §2.3 · p. 3

3 Method

AffordanceWAM jointly generates future scene representations and robot actions, using object-centric affordance streams to connect visual prediction with control. Its World and Action Experts share information through masked attention, while heterogeneous human and robot data provide complementary supervision.

3 Method

  • AffordanceWAM models future RGB, Scalar Affordance, Affordance Heatmap, and continuous actions through coupled World and Action Experts.The Experts use separate parameters and interact at corresponding layers through joint attention.Source: §3 · p. 4, §3.1 · p. 4, §3.2 · p. 5

3.1 Preliminaries

  • Flow matching trains velocity predictors to transport noisy future-World latents and continuous actions toward their data targets.Generation integrates learned velocities from noise at τ = 1 to data at τ = 0.Source: §3.1 · p. 4

3.2.1 World Expert.

  • The World Expert predicts RGB, Scalar Affordance, and Affordance Heatmap stream velocities from encoded RGB history and language.It is initialized from a pretrained video diffusion Transformer; history remains clean and is excluded from velocity targets.Source: §3.2.1 · p. 5

3.2.2 Action Expert.

  • The Action Expert predicts continuous action velocity from noisy action tokens, proprioception, language, and layer-wise RGB and Scalar Affordance features.Its inputs include clean historical RGB and evolving future RGB and Scalar Affordance features, but exclude Affordance Heatmap features.Source: §3.2.2 · p. 5

3.2.3 Masked Joint Self-Attention.

  • Masked Joint Self-Attention blocks Action-to-World information flow and prevents Affordance Heatmap features from reaching Action tokens, including indirectly.RGB and Scalar Affordance provide the World interface for action generation; direct RGB access remains available.Source: §3.2.3 · p. 5

3.3 Affordance-Aware World Representation

  • The future World combines RGB with Scalar Affordance interaction regions and Affordance Heatmaps that associate those regions with visual context.The streams share temporal grouping so attention can relate interaction structure to visual dynamics.Source: §3.3 · p. 6

3.3.1 RGB Future Representation.

  • A causal video encoder creates RGB history and future latents, with future slots representing appearance, object motion, and scene-state changes.Their temporal grouping defines the alignment used by the other two streams.Source: §3.3.1 · p. 6

3.3.2 Scalar Affordance Representation.

  • Scalar Affordance fields mark task-relevant objects and interaction regions without prescribing motor commands, while Heatmaps pair those regions with visual context.The scalar fields use shared human-and-robot semantics; the complementary RGB-space Heatmap target preserves the association between interaction regions and their objects.Source: §3.3.2 · p. 6, §3.3.3 · p. 6

4 Experiments

The experiments evaluate AffordanceWAM on simulation benchmarks, controlled ablations, and equal-data real-world manipulation. Results report gains over compared methods and show that the benefits of human videos depend on affordance supervision.

4.1 Experimental Setup

  • The evaluation uses RoboCasa task-specific adaptation, CALVIN ABC→D testing on unseen environment D, and real-world comparisons with equal fine-tuning data and trial conditions.The real-world comparison uses 50 trajectories per task and 15 trials per task for both methods.Source: §4.1 · p. 8

4.2 Simulation Benchmark Results

  • 69.4% RoboCasa average success is the highest among compared methods, exceeding Cosmos Policy by 2.3 percentage points.Table 1 reports average success on RoboCasa.Source: Table 1 · p. 8, §4.2 · p. 8
  • 4.22 average sequence length on CALVIN ABC→D exceeds the strongest included baseline’s 4.01, with AffordanceWAM also best across all reported metrics.Success rates for completing one to five consecutive tasks are 96.1%, 92.8%, 85.5%, 77.8%, and 69.5%, respectively.Source: Table 2 · p. 9, §4.2 · p. 9

4.3 Ablation Studies

  • With affordance, adding human videos raises RoboCasa success from 63.7% to 69.4% and CALVIN average sequence length from 3.91 to 4.22; without affordance, both metrics fall.Without affordance, the corresponding changes are 57.3% to 55.1% on RoboCasa and 3.75 to 3.68 on CALVIN, showing that human-video benefit depends on the representation.Source: §4.3 · pp. 9–10
  • Removing Scalar Affordance lets the World Expert largely discard interaction structure, while removing Affordance Heatmap weakens alignment between predicted interaction regions and visual objects.Removing Affordance Heatmap reduces RoboCasa success from 69.4% to 65.4% and CALVIN average sequence length from 4.22 to 3.99.Source: §4.3 · p. 10
  • RoboCasa success increases monotonically but non-uniformly as affordance-annotated human data scales, with fixed robot-data exposure and training updates.The first 60% of human data yields a 1.5-point gain; scaling from 60% to 100% adds 4.2 points, about 74% of the total improvement.Source: Table 4 · p. 10, §4.3 · p. 10
  • Opening the Affordance Heatmap-to-action path lowers RoboCasa success from 69.4% to 63.5% and CALVIN average sequence length from 4.22 to 3.85.The authors characterize masking as reducing an additional RGB-rich pathway while preserving direct access to RGB features.Source: §4.3 · p. 10

4.4 Real-World Evaluation

  • 74.7% macro-average success on basic real-world tasks exceeds Cosmos Policy’s 42.7% under the same adaptation budget.AffordanceWAM also improves drawer-closing success from 46.7% to 80.0%, three-fruit success from 26.7% to 53.3%, and flower-arrangement success from 6.7% to 20.0%.Source: Table 5 · p. 11, §4.4 · pp. 11–12

5 Conclusion

AffordanceWAM uses affordance as a shared interaction representation across human and robot data, with action generation grounded in robot trajectories. The conclusion reports strong results and says controlled ablations suggest benefits from human video when affordance is the shared predictive representation.

5 Conclusion

  • AffordanceWAM forecasts Scalar Affordance and Affordance Heatmap within the future World as a shared interaction representation across human and robot embodiments.Source: §5 · p. 12
  • A directionally coupled dual-expert architecture learns the World from human and robot data while grounding action generation exclusively in robot trajectories.Source: §5 · p. 12
  • AffordanceWAM achieves strong results on RoboCasa, CALVIN, and real-world tasks; controlled ablations suggest human-video benefits are pronounced when affordance is the shared predictive representation.Source: §5 · p. 12

Appendix

  • The supplement provides data and implementation details, training and sampling settings, seed-wise results, and additional quantitative and qualitative analyses.Source: Appendix · p. 16

A Data Preparation

Data preparation combines annotated egocentric human videos and robot trajectories, converts source annotations into shared affordance targets, and filters clips and trajectories for quality.

A.1 Sources and Splits

  • Human pre-training videos come from Ego4D-FHO, EPIC-KITCHENS VISOR, HOI4D, H2O, and MEC-CANO; robot trajectories come from RoboInter, BridgeData V2, InternData-A1, and RoboCasa365.Table 6 summarizes retained scale, sampling ratio, and downstream data budgets.Source: §A.1 · p. 16, Table 6 · p. 16

A.2 Affordance Target Construction

  • Target construction aligns source object and interaction annotations to sampled timestamps, then maps them to common task-relevant object and interaction-region labels.Source: §A.2 · p. 16
  • Scalar Affordance fields use Gaussian rasterization and per-frame peak normalization, with missing annotations interpolated over at most two frames and only visible support retained under occlusion.Affordance Heatmap targets are rendered from aligned RGB–Scalar Affordance pairs using a fixed renderer.Source: §A.2 · p. 16

A.3 Filtering and Quality Control

  • Human clips are filtered for excessive camera shake and invalid heatmaps, then manually reviewed; robot trajectories are checked for alignment, valid states and actions, and interaction annotations.Source: §A.3 · p. 16

B Model Implementation

The implementation specifies the pretrained backbone, expert dimensions, and prediction interface, then organizes video and affordance latents into temporal slots for past context and future prediction.

B.1 Backbone and Expert Dimensions

  • Table 7 specifies the pretrained backbone, Expert dimensions, and prediction interface, listing native hidden widths and joint-attention dimensions separately.Source: §B.1 · p. 16, Table 7 · p. 17

B.2 Temporal Latent Layout

  • The configured video encoder uses five temporal slots, with two forming context and three representing future frame groups 5–8, 9–12, and 13–16.The configuration uses P = 5, T = TA = 12, and dA = 7.Source: §B.2 · p. 17
  • Scalar Affordance uses three slots with the same future grouping, while Affordance Heatmap features are restricted to those future slots.Each slot contains spatial latent features rather than a single Transformer token.Source: §B.2 · p. 17

C Training Configuration

Training first calibrates the affordance codec and warms up the World Expert, then trains the Action Expert and joint World–Action generation. The configuration also specifies action refinement at inference and seed-based evaluation aggregation.

C Training Configuration

  • Stage I calibrates the Scalar Affordance codec and warms up the World Expert before Action training; Stage II freezes the World path and trains only the Action Expert for each target benchmark.During Action pre-training, the action-loss weight rises from zero to one, while fixed-World conditioning gives way to joint noisy-World/Action training.Source: C Training Configuration · p. 17

D Joint Generation and Action Refinement

E.1 Evaluation Units and Aggregation

  • Controlled RoboCasa and CALVIN results average three training seeds, with 1,200 episodes per RoboCasa seed and 1,000 five-task sequences per CALVIN seed; Tables 8 and 9 expand aggregate ablations.The reported variability is sample standard deviation across seeds; the tables align seed identifiers across controlled settings.Source: §E.1 · p. 18, Table 8 · p. 18, Table 9 · p. 18

E.2 Human-Data Scaling

The human-data scaling experiment reports seed-wise RoboCasa success rates, with three training runs and sample standard deviations underlying Table 4. Its experimental controls follow Section 4.3.

E.2 Human-Data Scaling

  • The experiment’s controls follow Section 4.3.Source: §E.2 · p. 18
  • Table 10 reports seed-wise RoboCasa success rates for the human-data scaling experiment.The table provides the three individual runs underlying Table 4.Source: §E.2 · p. 18, Table 10 · p. 19
  • The reported standard deviation is computed across three training seeds.Source: Table 10 · p. 19

F.1 Additional Architectural Ablations

Table 11 extends the architectural controls with alternative fusion designs, predicted-World post-training removal, and Affordance Heatmap removal. Fusion and masking controls match token count and trainable parameters under fixed data and update budgets.

F.1 Additional Architectural Ablations

  • Table 11 compares architectural ablations under the same data and update budget.Source: Table 11 · p. 19, §F.1 · p. 19
  • Fusion and masking controls additionally match token count and trainable parameters.Source: Table 11 · p. 19, §F.1 · p. 19
  • Late fusion communicates only through a final World-to-Action module, while the independent pipeline conditions a separately executed Action Expert on completed World prediction.Source: §F.1 · p. 19
  • The predicted-World post-training ablation removes Stage II exposure to World predictions, while the Affordance Heatmap ablation removes that auxiliary stream.The post-training ablation is distinct from final inference-time Action refinement.Source: §F.1 · p. 19
  • The open Affordance Heatmap→Action variant removes the block on Affordance Heatmap keys for Action queries.Source: §F.1 · p. 19

G.1 Qualitative Predictions and Failure Modes

The qualitative predictions include an aligned interaction-region localization and three failure types: spatial drift, RGB-copy collapse, and wrong-object localization. Further diagnostics distinguish representation errors from downstream control failures and long-horizon degradation.

G.1 Qualitative Predictions and Failure Modes

  • The aligned example localizes the interaction region on the intended object, while the other rows show spatial drift, RGB-copy collapse, and wrong-object localization.Source: §G.1 · p. 19, Figure 4 · p. 20
  • Spatial drift displaces the predicted interaction region from its object despite plausible RGB prediction.Source: §G.1 · p. 19
  • RGB-copy collapse lets appearance structure dominate the Affordance Heatmap, weakening interaction-specific localization.Source: §G.1 · p. 19
  • Wrong-object localization selects an object inconsistent with the instruction, even when the prediction is compact.Source: §G.1 · p. 19
  • Further diagnostics include failed control after plausible visual predictions and long-horizon degradation as localization worsens across successive future groups.These categories separate representation errors from downstream control failures.Source: §G.1 · p. 19

Related papers

Loading 2609.22332v3…