Source-linked AI summary
Flow as the Cross-Domain Manipulation Interface
Mengda Xu, Zhenjia Xu, Yinghao Xu, Cheng Chi, Gordon Wetzstein, Manuela Veloso, Shuran Song
TL;DR
Robots need manipulation learning that can use diverse data without costly real-world robot training. Im2Flow2Act uses generated object flow to connect cross-embodiment videos with simulated policy learning, and reports strong performance across varied real-world tasks. Its scope is limited by 2D flow ambiguity and the assumption that simulated and real object dynamics are consistent.
Problem
Cross-embodiment data suffers from embodiment gaps, while simulated robot data faces sim-to-real differences and costly task-specific environment construction.
Method
Im2Flow2Act generates task-specific object flow from cross-embodiment videos and conditions a policy trained on simulated exploration data to produce robot actions.
Results
Im2Flow2Act achieves the best performance among baselines in simulation and real-world evaluations, with an average 15% performance drop from simulation to real world.
Takeaways & Limitations
Object flow provides an interface connecting cross-embodiment and cross-environment data for manipulation of rigid, articulated, and deformable objects.
Takeaways & Limitations
The 2D flow representation is ambiguous for some 3D actions, and deployment assumes consistent object dynamics between simulation and the real world.
Abstract
from arXiv · showhide
We present Im2Flow2Act, a scalable learning framework that enables robots to acquire real-world manipulation skills without the need of real-world robot training data. The key idea behind Im2Flow2Act is to use object flow as the manipulation interface, bridging domain gaps between different embodiments (i.e., human and robot) and training environments (i.e., real-world and simulated). Im2Flow2Act comprises two components: a flow generation network and a flow-conditioned policy. The flow generation network, trained on human demonstration videos, generates object flow from the initial scene image, conditioned on the task description. The flow-conditioned policy, trained on simulated robot play data, maps the generated object flow to robot actions to realize the desired object movements. By using flow as input, this policy can be directly deployed in the real world with a minimal sim-to-real gap. By leveraging real-world human videos and simulated robot play data, we bypass the challenges of teleoperating physical robots in the real world, resulting in a scalable system for diverse tasks. We demonstrate Im2Flow2Act's capabilities in a variety of real-world tasks, including the manipulation of rigid, articulated, and deformable objects.
1 Introduction
Im2Flow2Act uses object flow as a shared interface between cross-embodiment demonstrations, simulated robot data, and real-world manipulation. Its two-part system generates task flows from videos and maps them to robot actions, supporting diverse object types without real-world robot training data.
- Cross-embodiment learning is hindered by embodiment gaps, while simulation faces sim-to-real differences and costly task-specific environment construction.
- Im2Flow2Act uses object flow to connect human or robot demonstrations with simulated robot data without requiring real-world robot training data.
- The flow generation network learns task planning from cross-embodiment videos, while the flow-conditioned policy learns from embodiment-specific simulated data.
- Object flow represents pose changes, articulations, and deformations, enabling one manipulation policy to cover rigid, articulated, and deformable objects.
- The system bridges cross-embodiment videos and cost-effective simulated robot data through object flows as a scalable manipulation interface.
- 81% average success was achieved across four real-world tasks without real-world robot training data.
2 Related Work
Related work studies cross-embodiment learning through visual, reward, affordance, pose, and translation methods, while flow-based manipulation has shown promise but remains task-limited. Because cross-embodiment actions are difficult to transfer, prior systems often still require in-domain robot data.
- Flow-based policies have addressed articulated objects and tools, but earlier approaches remain limited to specific tasks.
- Cross-embodiment research explores visual pretraining, reward learning, affordance extraction, hand-pose detection, and domain translation.
- Cross-embodiment data often lacks explicit actions or contains actions that are difficult to transfer across embodiment gaps.
- Many prior cross-embodiment methods therefore still require in-domain robot data.
3 Method
The method generates complete object-centric task flows from an initial frame and task description, then uses a flow-conditioned policy to produce robot actions. Training combines cross-embodiment videos with simulated exploration data while aligning current execution with the remaining task flow.
- Object flow serves as a unified interface that captures transferable object motion across embodiments and simulation-to-real environments.
- The flow generator detects the object, samples grid keypoints, generates future flow with AnimateDiff, and filters motion into an object-centric task flow.
- The policy is trained entirely on simulated exploration data generated by predefined primitive actions to learn actions-to-flow relationships.
- Generating a complete flow from the initial frame reduces embodiment-related visual gaps because the embodiment need not remain in contact with the object.
- Flow is compressed into a lower-dimensional latent space before generative modeling, with spatial dimensions downsampled by a factor of 8.
- The policy takes complete task flow, current keypoint state, and robot proprioception as inputs and outputs a sequence of robot actions.
- Temporal alignment predicts the remaining task flow from the complete flow and current state so action prediction follows task progress.
4 Evaluation
Im2Flow2Act was evaluated on four manipulation tasks spanning rigid, articulated, and deformable objects in simulation and the real world. It achieved the best baseline performance, while ablations showed that object-centric flow and alignment support cross-embodiment transfer and reliable action execution.
- Experiment Details: Evaluations covered pick-and-place, pouring, drawer opening, and cloth folding across rigid, articulated, and deformable objects.The study used demonstration-conditioned and language-conditioned execution in simulated and real-world settings.
- Overall Results: Im2Flow2Act achieved the best performance among baselines in both simulation and the real world, with an average real-world performance drop of 15% from simulation.The policy followed generated flow across the evaluated object types.
- Policy Comparison: Learning-based flow-to-action translation was necessary for accurate and safe manipulation, especially with articulated and deformable objects.Heuristic policies depended heavily on camera-view contact points and could push drawers incorrectly or trigger emergency stops.
- Object Flow: Im2Flow2Act outperformed ATM by an average of 30% across four tasks because it generates object flow rather than embodiment-dependent flow.ATM encounters out-of-distribution visual inputs when the deployment embodiment differs from the training embodiment.
- The alignment module is necessary: Alignment produced smoother execution and substantially improved real-world performance when human and simulated flows differed.Without alignment, the policy could select random or unnecessary trajectories, including sudden rotations.
- Limitations: The 2D-flow interface is ambiguous for 3D actions, and the assumed simulation-to-real consistency is especially limiting for deformable-object manipulation.The paper reports imprecise z-axis movement in pouring, difficulty with out-of-plane rotation, and a significant real-world performance drop in folding.
5 Conclusion
Im2Flow2Act provides a scalable route to learning diverse robot manipulation skills from cost-effective cross-domain data. Its reported real-world capability and cross-domain performance support object flow as a practical unifying interface, while simulation remains important for reducing real-world data collection demands.
- Conclusion: Im2Flow2Act enables robots to acquire diverse manipulation skills from cost-effective cross-domain data using object flow as a unifying interface.The framework targets learning without relying on real-world robot training data.
- Conclusion: The system outperformed all baselines across varied real-world manipulation tasks, demonstrating strong real-world manipulation capability.The conclusion presents this as evidence for scaling robotic manipulation skills from diverse data sources.
- Sim2Real Transfer: Sim-to-real policy learning is motivated by the cost and difficulty of collecting real-world robot data, but typically requires task-specific handling of the sim-to-real gap.Prior approaches include depth observation, domain randomization, knowledge distillation, and system identification.
B Additional Experiment
The additional experiment compares Im2Flow2Act with ATM for flow generation during robot deployment. Im2Flow2Act remains robust to cross-embodiment inputs, whereas ATM produces high-quality flows only under same-embodiment training.
- Im2Flow2Act generates high-quality object-centric flows despite the UR5 appearing in the visual input.ATM generates high-quality flows with same-embodiment training but noisy flows when trained on sphere demonstrations and deployed with UR5.
- ATM’s cross-embodiment flow quality degrades because its visual input becomes out-of-distribution during robot deployment.ATM predicts embodiment flow; replacing the training human arm or sphere with a robot arm and gripper creates an unseen embodiment.
- Im2Flow2Act avoids this failure by generating only object flow and using initial object keypoints as a strong condition for flow generation.Object-centric flow does not focus on the embodiment, supporting robustness across different embodiments at deployment.
- The complete-system comparison evaluates Im2Flow2Act and ATM under matched flow-generation and policy settings, with ATM’s performance dropping for cross-embodiment training.Im2Flow2Act retains strong capability when learning from cross-embodiment demonstrations, while ATM performs substantially better with same-embodiment demonstrations.
B.1 Long Horizon Task with Multiple Objects
The long-horizon experiment tests a sequence involving drawer opening, cube placement, and drawer closing with randomly initialized object positions. Im2Flow2Act generates and follows complete multi-object flows, achieving high success rates in both execution settings.
- Temporal sub-sampling enables compact flow generation for long-horizon tasks and improves robustness when demonstration and robot speeds differ.The same training and inference parameters are used as in the other tasks.
- The system generates complete object-centric flows for long-horizon tasks involving multiple objects, and robot execution closely follows them.Multiple object bounding boxes are obtained, keypoints are sampled proportionally to box area, and the keypoints are arranged into an H × W flow image.
- ATM performance suffers from out-of-distribution visual input in this multi-object setting.
B.2 Ablation Study on Initial 3D Keypoints
This section examines design choices affecting flow generation and policy execution, including initial 3D keypoints, motion filtering, pretrained initialization, and cross-embodiment deployment. The reported results support the importance of these components for robust object-centric flow control.
- Initial 3D keypoints improve policy performance when generated flow is used for language-conditioned execution.They help distinguish noisy keypoint movement near objects from movement belonging to the manipulated object, such as a drawer handle.
- The system requires 3D keypoints only initially because subsequent movement is represented relative to those points.Avoiding per-frame 3D keypoint acquisition keeps the real-world system simpler and more efficient.
- The flow-generation model is trained from human demonstrations for real-world deployment, while the manipulation policy is evaluated using simulated data.
- A pretrained Stable Diffusion U-Net yields faster convergence and lower flow-generation loss than training from scratch.The comparison is visualized in the training-loss curves, whose loss is not smoothed.
- Im2Flow2Act generates high-quality flow from cross-embodiment demonstrations, whereas ATM produces noisy flow under the same condition.The comparison uses complete ATM grid flow and filtered object flow, with ATM evaluated using 400 grid keypoints versus 128 object keypoints in Im2Flow2Act.
C.1 Simulated exploration Data
The simulated exploration dataset covers rigid, articulated, and deformable objects using predefined randomized robot behaviors. The resulting trajectories are converted into object flows through point tracking, filtering, and task-specific human-video processing.
- The simulation includes rigid objects, drawers, and cloth, with an UR5e robot exploring through predefined random heuristic actions.These environments provide exploration data for different object types and manipulation behaviors.
- Rigid-object exploration uses randomized cubic Bézier trajectories with sampled waypoints, target orientations, and terminal rotation or placement actions.The trajectory uses two consecutive 3D cubic Bézier curves and k = 16 sampled waypoints.
- Drawer exploration varies the selected drawer, contact point on its handle, and opening extent, while cloth exploration varies grasp corner and folding target.
- The dataset contains 4800 random exploration trajectories, from which object flows are constructed using grid keypoints, Tapir tracking, moving filters, and SAM filters.
- Human demonstrations cover pick and place, pouring, drawer opening, and cloth folding, with object keypoints sampled from initial-frame bounding boxes and tracked across 32 sampled frames.The demonstrations are recorded with a RealSense camera at 30 FPS.
D Ablation study
The ablation evaluates pretrained Stable Diffusion against training from scratch for flow generation, while other policy conditions remain fixed. Generated flow nearly matches ground-truth-flow performance on several tasks but substantially underperforms for cloth folding.
- Pretraining ablation: Pretraining had a minor impact on Im2Flow2Act’s final performance when evaluated with the same flow-conditioned policy.Both flow-generation networks used the pretrained autoencoder, while only the U-Net initialization differed.
- Flow quality: Generated flow produces success rates close to ground-truth flow for pick & place, pouring, and drawer opening.The comparison was conducted in simulation.
- Flow quality: Cloth folding success drops significantly because the policy must grasp a tiny attached cube that is not rendered during inference.Multimodal flow predictions, dataset randomness, and motion filtering can prevent precise cube grasping.
E Experimental Details
The real-world evaluation uses a calibrated UR5e setup, varied initial scenes, explicit success criteria, and comparisons between aligned and unaligned conditions and a heuristic policy.
- Evaluation setup: The UR5e executes 2.5 Hz end-effector positional commands with speed and table-clearance limits, using a RealSense D415 for depth and RGB observations.The end-effector speed is limited below 0.2 m/s and position remains at least 1 cm above the table.
- Evaluation protocol: Initial backgrounds are not matched to human demonstrations, while object positions follow roughly the demonstrations’ distribution.This protocol also tests flow-generation generalization.
- Success criteria: Success is defined with task-specific behavioral criteria for pick-and-place, pouring, drawer opening, and related real-world tasks.For pick-and-place, success requires stable mug placement and plate movement of no more than 5 cm.
- Baselines and comparisons: The evaluation compares Im2Flow2Act with and without alignment across 20 episodes per task and includes a heuristic policy based on matched human demonstrations.The heuristic obtains open-loop actions by estimating object pose transformations from human RGB-depth recordings.
F Training Details
Training and inference combine language, visual, and keypoint conditions with flow generation, temporal encoding, filtering, tracking, and simulated cross-embodiment demonstrations.
- Flow generation: The flow generator conditions on task text, the initial frame, and initial object keypoints to preserve scene and object relationships.Text is encoded with CLIP, while image patches and keypoints receive separate embeddings.
- Flow generation: Training uses 32×32×32 flow representations with 1024 keypoints over 32 steps and Stable Diffusion fine-tuning with LoRA.The decoder is fine-tuned for 400 epochs at a 5e−5 learning rate.
- Policy training: Policy samples contain proprioception, current keypoints, action sequences, and complete task flows, with 128 keypoints and 16-action horizons.Task flows include the first and last episode frames while using randomly sampled intermediate frames.
- Policy training: The policy encodes current flow states temporally and uses a diffusion action head with 50 training and 16 inference steps.The policy is trained for 500 epochs with AdamW at a 1e−4 learning rate.
- Simulation demonstrations: Sphere-agent demonstrations use predefined heuristic action primitives to mimic human demonstrations in simulation.Separate primitives are specified for pick-and-place, pouring, drawer, and folding tasks.
I Additional Limitation
The method assumes calibrated camera viewpoints with visible actions, and its flow abstraction and object-detection dependence constrain applicability and performance.
- Scope and assumptions: The system assumes camera viewpoints are calibrated between simulation and testing environments, with actions visible from the camera.The authors also note limits for dexterous in-hand manipulation and tasks whose state changes are not reflected in object motion.