PaperlayerView exact revision on arXiv

Source-linked AI summary

General Flow as Foundation Affordance for Scalable Robot Learning

Chengbo Yuan, Chuan Wen, Tong Zhang, Yang Gao

arXiv:2401.11439v2cs.ROcs.AIcs.CV

TL;DR

Real-world robot manipulation needs scalable learning and a prediction target that can use large datasets. The paper predicts language-conditioned 3D point trajectories from RGBD human videos as actionable guidance, achieving 81% average success in zero-shot transfer across 18 tasks and 6 scenes.

  • Problem

    The paper asks how to acquire real-world manipulation skills scalably using a prediction target that supports efficient, universal learning.

    Source: §1 · p. 1

  • Method

    The method predicts future trajectories of object points as language-conditioned 3D flow, trained directly from large-scale RGBD human videos.

    Source: §1 · pp. 1–2

  • Results

    81% average success was achieved in zero-shot human-to-robot transfer across 18 tasks and 6 scenes, without in-domain finetuning.

    Source: §1 · p. 2

  • Takeaways & Limitations

    General flow provides actionable guidance for zero-shot transfer across embodiments and object categories, including rigid, articulated, and soft bodies.

    Source: §1 · p. 2, §6 · p. 8

    The diversity and volume of the data are currently limited, restricting adequate guidance for complex tasks.

    Source: I Limitations · p. 25

Abstract

from arXiv · show

We address the challenge of acquiring real-world manipulation skills with a scalable framework. We hold the belief that identifying an appropriate prediction target capable of leveraging large-scale datasets is crucial for achieving efficient and universal learning. Therefore, we propose to utilize 3D flow, which represents the future trajectories of 3D points on objects of interest, as an ideal prediction target. To exploit scalable data resources, we turn our attention to human videos. We develop, for the first time, a language-conditioned 3D flow prediction model directly from large-scale RGBD human video datasets. Our predicted flow offers actionable guidance, thus facilitating zero-shot skill transfer in real-world scenarios. We deploy our method with a policy based on closed-loop flow prediction. Remarkably, without any in-domain finetuning, our method achieves an impressive 81\% success rate in zero-shot human-to-robot skill transfer, covering 18 tasks in 6 scenes. Our framework features the following benefits: (1) scalability: leveraging cross-embodiment data resources; (2) wide application: multiple object categories, including rigid, articulated, and soft bodies; (3) stable skill transfer: providing actionable guidance with a small inference domain-gap. Code, data, and supplementary materials are available https://general-flow.github.io

1 Introduction

The paper seeks a scalable robot-learning framework built around a prediction target that can use large human-video datasets while keeping the inference domain gap small. It proposes general flow as actionable motion guidance and reports zero-shot human-to-robot transfer across real-world tasks.

1 Introduction

  • The framework targets scalable robot learning by using large-scale human videos rather than relying on difficult-to-collect real-world robot data.The paper identifies scalable data with a small inference domain gap and a foundational prediction task as elements of generalization.Source: §1 · p. 1
  • General flow represents future trajectories of 3D points on an object, which a robot can follow as motion guidance.For opening a safe, the predicted door-point flow guides the robot’s opening motion.Source: §1 · p. 2
  • The model predicts dense flow from RGBD human videos, using scale-aware strategies and augmentations for embodiment occlusion and query-point sampling.These design choices address trajectory-scale variability and zero-shot robustness.Source: §1 · p. 2
  • 81% average success across 18 diverse tasks and 6 scenes was achieved with one model and no in-domain finetuning.The evaluation used a Franka-Emika robot and a heuristic policy based on closed-loop flow prediction.Source: §1 · p. 2

2 Problem Formulation

The formulation focuses on providing prior guidance for post-grasp movement, where predicting motion requires understanding both object geometry and trajectory scale. General flow encodes future trajectories of spatial query points under a task instruction.

2 Problem Formulation

  • A general method for prior guidance of post-grasp movement remains a challenge, beyond the extensively explored problem of robust functional grasping.The paper focuses on the post-grasp movement component of affordance.Source: §2 · p. 2
  • Accurate post-grasp prediction must capture object-motion geometry, such as a safe door rotating while points on its base remain static.Points nearer the rotation axis require shorter predicted trajectories than points farther away.Source: §2 · pp. 2–3
  • General flow is defined as the trajectories of spatial 3D query points over future timestamps, given an observation and task instruction.The formulation describes observations from any embodiment and trajectories over T future timestamps.Source: §2 · p. 3
  • The formulation is intended to apply across embodiments and rigid, articulated, and soft object categories.The term general flow emphasizes transfer from different embodiments, including human to Franka-Emika.Source: §2 · p. 3

3 Embodiment-Agnostic Flow Prediction

The section presents embodiment-agnostic, language-conditioned 3D flow prediction trained from RGBD human videos, with scale-aware modeling and robustness-oriented training. It evaluates flow prediction and real-world zero-shot transfer.

3 Embodiment-Agnostic Flow Prediction

  • The framework develops language-conditioned 3D flow prediction directly from large-scale RGBD human videos and is agnostic to specific embodiments.The flow labels represent future trajectories of points on objects of interest.Source: §3 · p. 3

3.1 General Flow Label Acquisition

  • Labels combine HOI4D data for rigid and articulated objects with collected clothes-folding videos, using pose-based or tracked-and-back-projected trajectories.The pipelines filter annotation noise and retain hand masks for potential training augmentation.Source: §3.1 · pp. 3–4

3.4 Evaluation

  • ScaleFlow achieves superior performance on every ADE and FDE metric with fewer parameters; real-world transfer spans 18 tasks across 6 scenes.The evaluation includes rigid, articulated, and soft objects.Source: Table 1 · p. 4, §3.4 · p. 5, Figure 5 · p. 6

3.2 Scale-Aware Flow Prediction

  • Total Length Normalization rescales trajectories to address per-point length variation, while the VAE predicts trajectory scale and normalized displacement separately.The ablation reports that TLN performs best among the compared normalization methods.Source: §3.2 · p. 4

3.3 Training Flow Prediction Model

  • Training combines trajectory, scale-regression, VAE KL-divergence, and cumulative-shift losses; hand-point dropout and query-distribution switching target occlusion and application variation.The augmentations are proposed to improve zero-shot generalization robustness.Source: §3.3 · p. 5

4 Zero-shot Real World Manipulation

The paper uses one closed-loop flow-prediction model with a heuristic policy for zero-shot real-world manipulation, reporting 81% average success across 18 tasks and 6 scenes. Its analyses examine segmentation robustness, novel shapes, grasp variations, and diverse scenes and directions.

4.1 Heuristic Policy with General Flow

  • The heuristic policy predicts flow for gripper-near query points, aligns the end-effector transformation to that flow with SVD, and executes it using an impedance controller.Scene point clouds and text instructions are also model inputs.Source: §4.1 · p. 6

4.2 Real World Experiment Setting

  • The evaluation covers 8 objects in 6 scenes and 18 tasks, with 10 trials per task and randomly varied, unseen backgrounds.All task and object categories except Box appear in training; Toy Car and Safe are the only objects that are not novel instances.Source: §4.2 · pp. 6–7

4 Zero-shot Real World Manipulation

  • 81% average success across 18 tasks and 6 scenes was achieved with one model for rigid, articulated, and soft objects.The evaluation uses a heuristic policy based on closed-loop flow prediction and requires no in-domain finetuning.Source: §4 · p. 6, Table 2 · p. 7, §4.3 · p. 7

4.3 Results and Analysis

  • Even with almost failed robot segmentation, the method predicts meaningful flow that facilitates task completion in a closed loop.The results also report that random hand-mask augmentation during training improves robustness to segmentation-map errors.Source: §4.3 · pp. 7–8
  • For the open task, the atypical Box reached 100% success versus 90% for Safe; the authors attribute this to Box allowing more trajectory deviation without losing the gripper on the door.The compared objects are shown in Figure 5(b), and the rates are reported in Table 2.Source: §4.3 · p. 8
  • The model completed tasks with different gripper states on Refrigerator and Drawer handles, and pushed a Toy Car in different directions to a certain extent across six scenes.The experiments varied movable-object direction and used scene-based prediction without requiring clean object segmentation.Source: §4.3 · p. 8

5 Related Work

The related-work discussion contrasts general flow with prior approaches to scalable, embodiment-agnostic robot learning and flow-based prediction.

5 Related Work

  • Prior approaches use visual pretraining, video-generated action signals, or simulator affordances, while some human-video methods require in-lab training or have unstable performance.The authors position 3D flow-based affordance as a route to reliable zero-shot solutions.Source: §5 · p. 8
  • Earlier flow methods either depend on embodiment-specific data or simulation, while cited 2D flow pretraining requires in-domain finetuning.The paper extends Seita et al. [7] with 3D flow prediction directly from RGBD human videos and reports stable zero-shot skill transfer.Source: §5 · p. 8

6 Conclusion

The conclusion presents General Flow as an affordance for scalable robot learning, trained from large-scale RGBD human videos and deployed for zero-shot human-to-robot transfer. The appendix overview points to further material on emergent properties, experiments, and framework limitations.

6 Conclusion

  • The paper introduces General Flow as Foundation Affordance and reports a language-conditioned flow model trained directly from large-scale RGBD human videos, deployed with a heuristic policy for zero-shot human-to-robot transfer.The authors describe the framework as combining scalability, wide application, and stable skill transfer.Source: §6 · p. 8

A Overview of Appendix

  • The appendix overview identifies emergent-property demonstrations, an ablation study, expanded real-world experiments, and discussion of limitations including data diversity, policy backbone, manual grasping, representation format, and heuristic policy.The overview says emergent properties are demonstrated after large-scale training and validated in a real-robot experiment.Source: A Overview of Appendix · p. 15

B Label Extraction Pipeline

  • The label-extraction pipeline section describes extracting general-flow labels directly from 3D human datasets or RGBD videos and points to Figure 6 for examples of data resources.Source: B Label Extraction Pipeline · p. 15

B.1 From HOI4D Datasets

The appendix describes how the paper sources and processes RGBD human-video data for general-flow labels, then details model architecture and training augmentations.

B.1 From HOI4D Datasets

  • HOI4D is the primary label source because of its relatively large scale and comprehensive 3D annotations, including RGBD images, camera parameters, object poses, segmentation masks, and action labels.The annotations support 4D human-object-interaction research.Source: §B.1 · p. 15

B.2 From Collected RGBD Videos

  • For soft-object coverage, the authors collected 30 RGBD rollouts of six clothing types for folding and extracted 605 clips; tracked 2D trajectories are projected into 3D flow labels.They cite existing HOI datasets' limited scale or semantic richness and the scarcity of soft-object data as motivation for collecting these videos.Source: §B.2 · p. 16

C.1 Architecture

  • The described ScaleFlow variants use PointNeXt backbones, CLIP text features, and a conditional VAE, with model width and encoder-decoder size varying by variant.The loss parameters emphasize scale prediction, with β1 = 25 and β2, β3 = 1.Source: §C.1 · p. 17

C.2 Augmentation Techniques

  • Hand-mask augmentation simulates absent, full, or partial hand-point occlusion, while query-point sampling alternates between random points and points clustered around an anchor.These augmentations target embodiment occlusion and variation in downstream query-point distributions.Source: §C.2 · p. 17

C.3 Training Details

Training details describe scale rebalance and data sampling, while the baseline section adapts 2D, affordance, and 3D models for comparison.

C.3 Training Details

  • Scale rebalance clusters flow trajectories by scale and resamples all but the largest cluster; the default temperature τ is 1.The stated motivation is to counter the bias toward stationary trajectories when most query points are static.Source: §C.3 · pp. 17–18
  • Training uses 1.5-second clips with three flow steps, split into 51,693 training, 6,950 validation, and 6,835 test clips without shared object instances.Each sample uses 2,048 scene points and 128 query points for training; validation samples 512 query points.Source: §C.3 · p. 18
  • Testing distributes 512 query points equally across object parts, but part labels are not used in validation or real-world testing.Source: §C.3 · p. 18

D Baselines

  • The baselines include ResNet, ViT, and R3M; VAT-MART’s trajectory branch; and FlowBot3D- and ToolFlowNet-style 3D models with PointNeXt.Aligned text features are incorporated into the baselines, and scale rebalance, HMA, and QPS are applied for fair comparison.Source: D Baselines · p. 18
  • Table 3 reports test-set ADE and FDE, with ADE-H and FDE-H evaluating inputs that include hand points; the caption reports robustness with appropriate augmentations.Source: Table 3 · p. 19

E.1 Properties Analysis

The section examines general flow’s emergent prediction properties, validates language control and scale adaptation on a real robot, and reports dataset ablations.

E.1 Properties Analysis

  • With appropriate augmentations, the model is robust to hand points in its input, evaluated using ADE-H and FDE-H.Source: §E.1 · p. 19
  • Different instructions change predicted flows for identical point clouds, while predictions retain correct trends under severe label noise and adapt trajectory scale to spatial distance.Source: §E.1 · p. 19

E.2 Real-Robot Validation

  • With initial states of Drawer, Refrigerator, and Mug kept consistent, varying instruction verbs enables correct semantic actions; for identical “putdown Mug” tasks, predicted scale averaged 19.04cm at higher and 5.94 cm at lower Mug positions.Source: §E.2 · p. 20, Figure 9 · p. 20

F.1 Dataset Experiment Setting

  • Except when robustness augmentations were removed, ablated variants degraded prediction performance; hand-mask augmentation significantly improved in-domain prediction.Source: §F.1 · p. 21, Table 4 · p. 21

F.2 Real-Robot Ablation Study

The section reports real-robot augmentation ablations, describes the evaluation environment and deployment policy, and identifies a segmentation latency bottleneck.

F.2 Real-Robot Ablation Study

  • Each augmentation significantly contributes to robust zero-shot execution; hand masking matters under heavy occlusion, while QPS is influential because PointNeXt couples query-feature extraction.Source: §F.2 · p. 21

G.1 Real-World Environment Setting

  • Eight objects spanning rigid, articulated, and soft bodies populate 18 tasks across 6 scenes; movable objects vary in position and orientation.Source: §G.1 · p. 22, Figure 10 · p. 22

G.2 Real-World Baseline

  • The real-world baseline provides only 2D guidance, with weak motion-direction generalization and embodiment-occlusion disturbances, so its trajectories were not used for robot trials.Source: §G.2 · pp. 22–23, Figure 11 · p. 23

G.3 Development of the Robotic System

  • Deployment predicts flow from gripper-near query points and follows an SVD-aligned motion closed-loop; paths shorter than 5 cm are consolidated and scaled to 5 cm.Source: §G.3 · p. 24

G.4 Heuristic Policy Derivation

  • The heuristic policy fits a 6DoF rotation and translation between adjacent point clouds by weighted SVD, assigning greater weight to query points nearer the gripper.Source: §G.4 · p. 24

G.5 Inference Latency

  • FastSAM segmentation accounts for 85.7% of average pipeline latency in 10 measurements of the “open(grasp) Refrigerator” task.Without FastSAM, point-cloud stream acquisition also presents substantial room for improvement.Source: §G.5 · p. 24

G.6 Failure Case Analysis

The section analyzes low-success robot tasks and situates the approach among real-world general robot learning methods.

G.6 Failure Case Analysis

  • “Push Toy Car” succeeds in 50% of trials; the authors identify direction semantics and small-object occlusion as challenges, suggesting more data and a wrist camera.Source: §G.6 · p. 25
  • “Open Drawer” has 30% grasping and 40% pulling success; mixed prismatic and revolute training structures bias predictions toward rotation, while the handle can slip.Source: §G.6 · p. 25
  • “Open Laptop” succeeds in 50% of trials; its thin lid often produces poor or incorrect RGBD point clouds, for which fused multi-view clouds are suggested.Source: §G.6 · p. 25

H Related Works of Real-World General Robot Learning

I Limitations

The authors identify limits in dataset scale, model expressivity, grasping automation, and the zero-shot heuristic policy. The codebases section credits libraries and prior implementations used in the work.

I Limitations

  • Limited data diversity and volume restrict the model’s ability to guide complex tasks.The authors suggest larger RGBD datasets or RGB datasets paired with depth estimation as possible ways to address this limitation.Source: I Limitations · p. 25
  • The Conditional VAE may lack expressive power in more challenging scenarios, such as larger and more diverse datasets.The authors identify diffusion as a potential alternative.Source: I Limitations · p. 25
  • Future work could automate manual grasping and extend object-centric affordance toward universal keypoints motion.The authors cite heuristic and automated grasping methods as options for replacing manual grasping.Source: I Limitations · p. 26
  • The initial ICP-based zero-shot heuristic may limit task completion, particularly for contact-rich manipulation.The authors suggest few-shot policy finetuning or using general flow as an action constraint for sample-efficient reinforcement learning.Source: I Limitations · p. 26

J Codebases

  • The training framework and 3D backbone are based on Qian et al. [20], while implementations of models and the VAT-MART baseline draw on Bao et al. [27], Nair et al. [25], and Wu et al. [10].Source: J Codebases · p. 26
  • ROS perception and data-stream acquisition are based on Shridhar et al. [84], and the SVD transformation solver is inherited from Zhong et al. [85].Source: J Codebases · p. 26
  • The work credits Doexys for the end-effector impedance controller and prior methods for point-cloud processing, HOI detection, segmentation, and point tracking.The cited sources are Zhu et al. [31], Liu et al. [15], Shan et al. [16], Zhao et al. [17], and Karaev et al. [18].Source: J Codebases · p. 26
Loading 2401.11439v2…