PaperlayerView exact revision on arXiv

Source-linked AI summary

A4A: Cross-Embodiment Transfer of Action-Oriented 4D Affordances from Human Demonstrations

Yifan Han, Litao Liu, Yuqi Gu, Ye Lu, Hanqing Wang, Sidney Wai, Ishaan Myrie, Qi Zhang, Jingjin Yu, Gen Li

arXiv:2609.05892v1cs.RO

TL;DR

Human demonstrations offer rich manipulation knowledge, but it remains unclear which information can guide robot control across embodiments, while conventional affordances mainly indicate where to interact. A4A represents task-conditioned motion as trajectories of interaction-relevant 3D points and uses trajectory prediction to pretrain robot policies. The paper reports consistent downstream gains across VLA policies in simulation and real-world tasks.

  • Problem

    It remains unclear which information in human demonstrations can guide robot control across embodiments, while static affordance representations primarily indicate where interaction may occur.

    Source: §1 · pp. 1–2

  • Method

    A4A uses language-conditioned trajectories of interaction-relevant 3D points as a pretraining target, then adapts policies to robot control through their native state and action interfaces.

    Source: §1 · p. 2, §3 · p. 3, §3.3 · p. 5

  • Results

    Average success rate improved for every evaluated VLA policy, with a 29.0 percentage-point gain for Octo.

    Source: §4.2 · p. 6

  • Takeaways & Limitations

    The results establish action-oriented 4D affordances as an effective cross-embodiment representation for transferring manipulation knowledge from human demonstrations to robot control.

    Source: §6 · pp. 7–8

    The current A4A formulation remains limited on large non-rigid deformation, severe slippage or repeated contact switching, and complex bimanual coordination.

    Source: §5 · p. 7

Abstract

from arXiv · show

Human demonstrations contain rich manipulation knowledge, but it remains unclear what information can be transferred effectively to robot control. Existing affordance representations are typically formulated as 2D masks, 3D regions, contact points, or actionability scores, and therefore primarily identify where interaction may occur. However, effective manipulation also requires modeling how interaction-relevant geometry evolves during task execution. To bridge this gap, we introduce action-oriented 4D affordances, which represent the language-conditioned future trajectories of interaction-relevant 3D points. These trajectories capture task-conditioned geometric evolution rather than embodiment-specific actions, enabling transferable interaction priors across humans and robots. Based on this representation, we construct a large-scale action-oriented 4D affordance dataset from existing human--object interaction video data and complementary RGB-D demonstrations, and introduce A4A, an affordance-to-action framework that uses 4D affordance trajectory prediction to pretrain robot policies before manipulation fine-tuning. Experiments in both simulation and the real world validate the effectiveness of A4A, showing that pretraining with action-oriented 4D affordance data consistently improves the manipulation performance of diverse VLA policies. These results establish action-oriented 4D affordances as an effective cross-embodiment representation for transferring manipulation knowledge from human demonstrations to robot control.

1 Introduction

The introduction argues that human demonstrations need to convey task-conditioned interaction motion, not merely spatial affordances or embodiment-specific actions. It introduces action-oriented 4D affordances and A4A as a way to transfer that geometric supervision into robot policies.

1 Introduction

  • Human videos contain diverse manipulation cues, but it remains unclear which can directly guide robot control while transferring across embodiments.Human motion may be embodiment-specific, while object identity alone provides only indirect control supervision.Source: §1 · pp. 1–2
  • Static affordances locate where interaction may occur but provide limited guidance on how an interaction should unfold.For example, a handle does not specify pulling displacement, and a cup rim does not define pouring motion.Source: §1 · p. 2
  • Human demonstrations expose task-conditioned motion of interaction-relevant 3D geometry, describing physical state transitions rather than human motor commands.Examples include outward translation for pulling a drawer, rotation for opening a cap, and short normal motion for pressing a button.Source: §1 · p. 2
  • Action-oriented 4D affordances are language-conditioned trajectory fields over 3D query points whose initial positions ground interaction and future paths specify its operation.The temporal representation connects perception with control by describing how interaction geometry evolves over time.Source: §1 · p. 2
  • A4A uses identity-preserving 3D point trajectories from human videos as an intermediate VLA pretraining target, then adapts policies through their native interfaces.The paper reports consistent downstream gains across policy families, strong RGB-only performance against methods using 3D or 4D observations, and more effective transfer than generic scene-wide 4D pretraining in an architecture-controlled comparison.Source: §1 · p. 2

2 Related Work

Related work spans affordance learning, robot learning from human demonstrations, and embodiment-agnostic manipulation representations. The paper positions its contribution as task-conditioned 4D motion of interaction-relevant geometry rather than spatial affordance cues or generic scene motion.

VLA and Affordance Learning

Learning Robot Manipulation from Human Demonstrations

Embodiment-Agnostic Representations for Manipulation

  • 2D tracks and flow-like signals lack metric 3D structure and are viewpoint-sensitive; structured 3D/4D approaches model spatial and temporal motion, while this work focuses on interaction-relevant points.The cited structured approaches include point trajectories, scene flow, and physically grounded dynamics (Yuan et al. 2024a; Han et al. 2026; Huang et al. 2026).Source: §2 · p. 3

3 Method

A4A transfers manipulation knowledge by pretraining VLA policies to predict task-conditioned trajectories of interaction-relevant 3D points, then adapting the shared representation to robot control. Its method rests on short-horizon geometric correspondence, a human-demonstration affordance dataset, and restoration of each policy’s native control interfaces.

3.3 Affordance-to-Action Representation Transfer

  • A4A pretrains each VLA on future 4D point motion, then restores its native state and action interfaces for robot finetuning while retaining the shared representation.The same internal module models interaction-centric point motion during pretraining and embodiment-specific actions during finetuning.Source: §3 · p. 3, §3.3 · pp. 4–5

3 Method

  • A4A uses future trajectories of interaction-relevant 3D points as its pretraining target, then transfers the learned representation through each policy’s native control interfaces.The representation transfer is guided by the short-horizon geometric alignment between point motion and end-effector motion.Source: §3 · p. 3, Figure 2 · p. 4

3.1 Geometric Premise for Cross-Embodiment Transfer

  • Over short horizons, end-effector displacement and attached-point motion encode the same transition; broader interaction-relevant geometry is treated as a local rigid or quasi-rigid approximation.Human trajectories provide pretraining supervision without robot control commands, with embodiment-specific control learned during finetuning.Source: §3.1 · p. 3

3.2 4D Affordance Data Construction

  • The dataset contains over 80K interaction clips, combining existing human–object data with RGB-D demonstrations that extend coverage to underrepresented operations.Query points on the manipulated object, tool, or contacted part are tracked and back-projected into 3D to form task-conditioned trajectories.Source: §3.2 · pp. 3–4

4 Experiments

Across simulation and physical manipulation, A4A improves diverse VLA policies; matched-architecture comparisons also favor action-oriented supervision over generic scene-wide 4D tracks.

4.1 Experimental Setup

  • Evaluation spans ten LIBERO-Object tasks, four RLBench tasks, and six physical Piper skills, with paired real-world comparisons holding downstream training and evaluation fixed.LIBERO-Object includes five VLA policy families; each is trained on 500 demonstrations and evaluated for 50 episodes per task.Source: §4.1 · p. 5, Table 3 · p. 7

4.2 Experiment Analysis

  • On RLBench, the RGB policy rises from 39.0% to 61.0% average success, highest overall; it leads on meat off grill and slide block and ties on turn tap.The downstream policy uses no explicit 3D observations.Source: Table 2 · p. 6, §4.2 · p. 6
  • A4A raises average success by 29.0 percentage points for Octo, 10.0 for OpenVLA, 0.6 for OpenVLA-OFT, 10.0 for π0, and 2.0 for π0.5.The gains span five policy families with different action-generation paradigms.Source: §4.2 · p. 6, Table 1 · p. 6
  • On physical Piper tasks, mean success across policies rises from 56.7% to 72.5%; pouring rice and water improve, requiring coordinated translation and orientation changes.The reported evaluation covers six physical tasks.Source: Table 3 · p. 7, §4.2 · p. 7
  • With architecture and training conditions matched, action-oriented supervision raises ARM4R average success from 41.0% to 60.0%, and turn tap from 28.0% to 60.0%.The comparison is against generic scene-wide 4D point-track initialization.Source: Figure 4 · p. 7, §4.2 · p. 7

5 Limitations

A4A's current instantiation is limited for interactions involving large deformation, severe slippage, repeated contact switching, or complex bimanual coordination.

5 Limitations

  • Large non-rigid deformation and severe slippage remain challenging for the current A4A instantiation.Source: §5 · p. 7
  • Repeated contact switching and complex bimanual coordination are also outside the formulation's reliable scope.Source: §5 · p. 7
  • Multi-turn cap opening requires release, regrasping, and contact re-establishment, which the current transfer formulation does not model.Training with affordances constructed from robot motion is proposed as a complementary future direction, not part of the current setting.Source: §5 · p. 7

6 Conclusion

A4A represents human manipulation as language-conditioned future trajectories of interaction-relevant 3D points and uses them to pretrain robot policies. Experiments report consistent downstream improvements across VLA families, simulated benchmarks, and real-world tasks.

6 Conclusion

  • Experiments across five VLA families, simulated benchmarks, and real-world tasks show consistent downstream improvements.The conclusion also reports that an architecture-controlled comparison further validates interaction-centric 4D supervision.Source: §6 · p. 7
  • A4A uses point-trajectory prediction to initialize VLA policies before adapting them to their native state and action interfaces.The framework represents human manipulation as language-conditioned future trajectories of interaction-relevant 3D points.Source: §6 · p. 7
  • The results establish action-oriented 4D affordances as an effective cross-embodiment representation for transferring manipulation knowledge to robot control.The approach is motivated by short-horizon geometric correspondence between human interaction and robot end-effector motion.Source: §6 · pp. 7–8

Supplementary Material for

The supplied passage gives the title of the supplementary material.

Supplementary Material for

from Human Demonstrations

The corpus combines existing human–object interaction videos with complementary RGB-D demonstrations, then reconstructs interaction-centric 3D point trajectories. Its examples span varied manipulation motions and geometric transitions.

A.1 Corpus Composition and Affordance Coverage

  • The corpus combines HOI4D and EPIC-KITCHENS with in-house RGB-D demonstrations, totaling over 80K clips, including approximately 30K in-house recordings.Its public portion supplies naturally occurring interactions, while the in-house subset expands coverage of less frequent or geometrically incomplete motions.Source: §A.1 · p. 10
  • Coverage includes common actions and less frequent tasks, spanning translation-dominant, rotation-dominant, articulated, tool-mediated, and object-to-object interactions.Examples include opening, pulling, pressing, pouring, cutting, hanging, sweeping, and lid removal.Source: §A.1 · p. 10

A.2 Annotation and Trajectory Reconstruction

  • Demonstrations are normalized into interaction-centric point trajectories by localizing and segmenting the relevant object region, then sampling and identity-preservingly tracking query points.When direct 3D data are unavailable, image tracks are lifted using estimated depth, camera intrinsics, and camera-motion compensation; unreliable tracks are removed.Source: §A.2 · p. 10
  • The in-house RGB-D pipeline backprojects verified tracks through calibrated observations to produce metric 4D trajectories that retain translation and rotation.It uses refined depth, task-conditioned localization and segmentation, and identity-preserving tracking.Source: §A.2 · p. 10

A.3 Qualitative Visualization

  • Representative examples visualize placement, pickup, articulated opening and closing, drawer pulling, pouring, cutting, sweeping, and hanging across varied geometric transitions.The transitions range from short translations to large orientation changes and tool-mediated interactions.Source: §A.3 · p. 10

A.4 Data Independence

Human pretraining data and robot training and evaluation data are collected independently, sharing high-level manipulation categories but no object instances, scenes, or paired demonstrations.

A.4 Data Independence

  • The human and robot datasets share no object instances, scenes, or paired demonstrations, overlapping only in high-level manipulation categories.Real-robot training and evaluation objects do not appear in the in-house human demonstrations, making the setup a test of transfer beyond instance- or scene-level matching.Source: §A.4 · p. 10

B A4A across Five VLA Action Paradigms

A4A adapts 4D affordance pretraining to five VLA policy families while retaining each policy’s native prediction pathway and objective family.

B.1 Unified Affordance-to-Action Transfer

  • Colored point flows show displacement from the current timestep to the next, while their color gradient indicates temporal evolution; qualitative examples compare predicted and ground-truth motion targets.The examples cover closing an oven, cutting fruit, dragging a drawer, and pouring between cups.Source: Figure 5 · p. 11, Figure 6 · p. 13
  • Point-motion targets encode short-horizon trajectory residuals relative to constant-velocity extrapolation, excluding invalid or occluded tracks from the loss.The residual parameterization captures interaction-induced near-future motion without a long-horizon forecasting objective and assigns zero residual to locally constant-velocity motion.Source: §B.1 · p. 11

B A4A across Five VLA Action Paradigms

  • A4A preserves the native prediction pathway and objective family across five VLA policy families, changing stage-specific state and output interfaces.The families use discrete autoregressive generation, continuous regression, diffusion, end-to-end flow matching, or hierarchical flow matching.Source: B A4A across Five VLA Action Paradigms · pp. 10–11, Table 5 · p. 12

B.2 Discrete Autoregressive Generation: OpenVLA

  • OpenVLA quantizes normalized point-residual coordinates into tokens, predicts them autoregressively, then transfers its learned decoder and output head to native action-token fine-tuning.The point-specific input interface is removed during robot fine-tuning.Source: §B.2 · p. 12

B.3 Continuous Regression: OpenVLA-OFT

  • OpenVLA-OFT predicts continuous point residuals with its native L1 or MSE objective, then retains and jointly optimizes the decoder for robot action chunks.A point-state projector is replaced by the native proprioceptive interface during robot fine-tuning.Source: §B.3 · p. 12

B.4 Diffusion Generation: Octo

  • Octo replaces robot-action chunks with future point residuals as the diffusion target while retaining its block-causal Transformer, noise conditioning, and DDPM denoising objective.During fine-tuning, the original action dimensionality and conditioning interface are restored, with the Transformer initialized from 4D pretraining.Source: §B.4 · p. 12

B.5 Flow Matching: π0 and π0.5

  • For π0 and π0.5, A4A transfers through the low-level flow-matching action expert, while π0.5’s high-level semantic subtask pathway remains frozen.Point residuals serve as the clean flow endpoint; during fine-tuning, point projections are replaced by native state and action projections.Source: §B.5 · pp. 12–13

B.6 RLBench Policy Instantiation

  • The RLBench instantiation retains a Qwen3.5-4B causal Transformer, replacing the point projector with proprioceptive and action heads for jointly finetuned robot control.Both stages use context length 16, matching the architecture-controlled comparison with ARM4R.Source: §B.6 · p. 13

B.7 LIBERO Evaluation Protocol

The LIBERO-Object evaluation uses a common rollout protocol while retaining policy-specific observation and inference settings.

B.7 LIBERO Evaluation Protocol

  • All five policy families are evaluated on LIBERO-Object with their native observation and inference configurations retained.The corresponding base and A4A-instantiated policies use the same policy-specific settings.Source: §B.7 · p. 13
  • Each task uses 50 rollouts from the same official initial states, the benchmark’s task-specific success predicate, and a 280-step maximum horizon.Source: §B.7 · p. 13
  • OpenVLA-OFT receives proprioceptive state and rank-32 LoRA, while π0 and π0.5 use one observation frame and retain their native action-chunk horizons.Source: §B.7 · p. 13

Related papers

Loading 2609.05892v1…