PaperlayerView exact revision on arXiv

Source-linked AI summary

Fetch My Beer: Synthetic-to-real Hierarchical Policy for Smooth Pick-and-place

Yingyue Li, Chenyangguang Zhang, Ruida Zhang, Bowen Fu, Guangyao Zhai, Xiangyang Ji

arXiv:2609.18119v1cs.RO

TL;DR

Stable liquid transport requires trajectory-level control that suppresses sloshing and spillage, while existing learning pipelines face data-collection, simulation-cost, and multi-scale control challenges. The paper combines physics-validated synthetic data with hierarchical latent diffusion control. Evaluations report improved success, stability, and smoothness over representative baselines, including zero-shot real-world transfer without real-world demonstrations.

  • Problem

    Stable liquid transport requires maintaining object dynamics throughout execution, but fluid simulation is costly for reinforcement learning, teleoperation can induce sloshing, and flat policies struggle with multi-scale control.

  • Method

    The framework generates stability-aware synthetic demonstrations and uses a hierarchical controller that predicts SE(3) sub-goals before latent diffusion generates dense action chunks.

  • Results

    The framework improves success rates, trajectory stability, and motion smoothness in simulation and the real world compared with representative baselines, with zero-shot sim-to-real transfer.

  • Takeaways & Limitations

    Physics-validated demonstrations and hierarchical latent diffusion support dynamically stable and smooth liquid-container manipulation across simulation and real-world settings.

Abstract

from arXiv · show

Many real-world robotic applications require dynamically sensitive manipulation, where success depends not only on reaching a target state but on maintaining stable object dynamics throughout execution. We study the stable transport of liquid-filled containers, where a robot must move objects to target locations while suppressing sloshing and preventing spillage. Unlike conventional pick-and-place, this task imposes stringent requirements on motion smoothness and trajectory-level stability, exposing clear limitations in existing systems. Specifically, fluid simulation remains too costly for online reinforcement learning; human teleoperation introduces unintended accelerations that induce sloshing during imitation learning; and current policy pipelines optimize for task completion rather than dynamic stability. We propose a synthetic-to-real framework coupling physically validated data generation with a hierarchical, diffusion-based controller. The scalable data pipeline synthesizes grasps, filters unstable poses via a vision-language model, and validates transport trajectories through fluid simulation. The policy is organized with a high-level module that translates language and visual observations into SE(3) control targets, and a latent diffusion controller that first plans efficiently in a compact latent space and then decodes dense action chunks, enabling the high control frequency needed for smooth and stable motion. Extensive experiments show our system outperforms state-of-the-art manipulation policies in transport smoothness and dynamic stability. Our project page: https://fetch-my-beer.github.io/

I. INTRODUCTION

Dynamically sensitive liquid transport requires smooth, stable motion rather than only correct final positioning, but existing learning systems struggle with fluid-aware data and multi-scale control. The paper addresses this with physically validated synthetic demonstrations, hierarchical latent diffusion control, and stability-focused evaluation.

  • Liquid transport requires suppressing sloshing and spillage throughout execution, because abrupt accelerations, jitter, or excessive tilt can cause failure despite correct final positioning.
  • Existing systems face costly fluid simulation in reinforcement learning, unstable human teleoperation data for imitation learning, and difficulty balancing semantic reasoning with smooth high-frequency control.
  • The proposed framework combines scalable synthetic demonstrations filtered by grasp feasibility, smooth planning, and fluid simulation with a hierarchical diffusion-based controller.
  • The high-level module predicts task-relevant SE(3) sub-goals from language and multi-view observations, while the low-level latent diffusion policy generates dense action chunks for high-frequency execution.
  • The benchmark evaluates task success, trajectory smoothness, orientation stability, and worst-case tilt across in-domain and out-of-domain settings.

II. RELATED WORK

Prior work advances manipulation through model-based liquid control, behavior cloning, and large vision-language-action models, but dynamically stable transport remains challenging in unstructured settings.

  • A. Dynamically Sensitive Manipulation: Model-based liquid-transport methods reduce oscillation but often rely on accurate fluid priors, handcrafted controllers, or fixed robot configurations.
  • A. Dynamically Sensitive Manipulation: Dynamically sensitive manipulation requires both geometric accuracy and stable physical evolution, motivating learning-based approaches that explicitly address stability.
  • B. Learning-based Manipulation: Behavior cloning and large VLA models map visual and language inputs to actions, but dense action prediction can be computationally expensive and temporally inconsistent.

C. Synthetic Data Generation

The synthetic-data pipeline scales demonstration collection while targeting semantic consistency, kinematic feasibility, smoothness, and dynamic safety. It combines diverse grasp proposals with scene-aware filtering and trajectory validation.

  • Existing synthetic pipelines mainly emphasize rigid-body collision avoidance and goal reaching, which is insufficient for dynamically sensitive transport.
  • The pipeline samples diverse scenes and objects, generates 6-DoF grasp candidates, plans grasp-to-place motions, and filters trajectories using tilt screening and fluid simulation.
  • The resulting demonstrations satisfy semantic consistency, kinematic feasibility, and trajectory-level dynamic stability for sim-to-real learning.
  • Multiple grasp hypotheses cover varied approach directions and contact regions, supporting stable downstream motion rather than selecting solely for reachability or force closure.

B. VLM-based Filtering

A vision-language model filters grasp proposals for scene-specific feasibility before motion planning, while complementary tilt screening and physics simulation reject trajectories that remain dynamically unsafe.

  • B. VLM-based Filtering: The VLM rejects grasps that are visually implausible or unsuitable for transport, including table collisions, infeasible approaches, unstable contacts, and unfavorable object poses.
  • B. VLM-based Filtering: VLM confidence ranks the surviving grasp candidates before trajectory planning, combining geometric proposal coverage with visual and commonsense feasibility assessment.
  • C. Smooth Path Planning with Stability-aware Filtering: cuRobo generates smooth grasp-to-place trajectories through inverse kinematics and trajectory optimization from filtered grasps and sampled placement poses.
  • C. Smooth Path Planning with Stability-aware Filtering: The hierarchical policy predicts SE(3) sub-goals, denoises action sequences in latent space, and decodes dense action chunks for smooth high-frequency execution.
  • C. Smooth Path Planning with Stability-aware Filtering: Tilt screening provides a lightweight preliminary filter, while PBD fluid simulation removes geometrically successful trajectories that cause excessive oscillation or spillage.

IV. HIERARCHICAL LATENT DIFFUSION POLICY

The hierarchical policy separates task-level pose prediction from high-frequency trajectory generation, using latent diffusion to produce smooth actions efficiently. It conditions motion generation on multimodal temporal context and applies denoising and temporal ensembling to improve execution continuity.

  • A. High-level Keyframe Pose Prediction Policy: The high-level policy maps language and multi-view observations to SE(3) keyframe poses, while the low-level policy converts these sparse goals into dense trajectories.This decomposition separates semantic reasoning from high-frequency motion generation.
  • B. Low-level Latent Diffusion Policy: Latent diffusion compresses dense action chunks into lower-dimensional temporal and feature latents, generates motion plans there, and decodes them into executable trajectories.The compact latent space reduces inference latency relative to raw-action diffusion while retaining dense trajectory reconstruction.
  • B. Low-level Latent Diffusion Policy: The low-level controller is conditioned on visual observations, proprioceptive history, and the high-level goal pose to adapt actions while preserving motion consistency.Short-horizon RGB observations provide recent visual feedback, whereas longer proprioceptive histories capture motion dynamics and trajectory evolution.
  • B. Low-level Latent Diffusion Policy: During training, a conditional 1D U-Net predicts noise in perturbed latent action sequences, using the diffusion step and conditioning context as inputs.The latent diffusion policy uses a standard noise-prediction objective with a cumulative noise schedule.
  • B. Low-level Latent Diffusion Policy: At inference, iterative latent denoising produces an action sequence that a decoder expands into dense chunks, while temporal ensembling reduces discontinuities between overlapping predictions.The resulting procedure supports high-frequency control and improves execution smoothness.

A. Experimental Setup

The benchmark evaluates liquid transport across distribution shifts using task-success, stability, and smoothness metrics, with standardized baselines and evaluation protocols.

  • Evaluation Protocol: The benchmark tests in-domain and three out-of-domain splits, including unseen cups, unseen scenes, and their compound shift.Each split uses 50 configurations and three random seeds with fixed initializations.
  • Baselines: Comparisons include a shared-high-level RVT-2 + DP baseline and π0.5 models fine-tuned with few-shot or full training data.
  • Metrics: Success Rate (SR, %) counts trials that grasp, transport, and place the cup while keeping fluid loss below 5%.
  • Metrics: Orientation Error (OE, rad · s) measures cumulative deviation from upright orientation during execution, with lower values indicating steadier transport.
  • Metrics: Maximum Tilt Angle (MTA, ◦) captures worst-case transient cup deviation that cumulative orientation error may miss.
  • Metrics: Trajectory Smoothness (TS, m2/s6) measures average squared end-effector jerk, where lower values indicate fewer abrupt acceleration changes and less sloshing risk.

B. Simulation Evaluation

Simulation results show that the proposed hierarchical latent-diffusion policy improves stable transport in-domain and generalizes best under object and scene distribution shifts.

  • In-domain Evaluation: 10.49% higher in-domain success rate than RVT-2 + DP accompanies lower OE, MTA, and TS for the proposed method.
  • Baseline Comparison: VLA baselines perform substantially worse, and full-data π0.5 fine-tuning remains far below hierarchical methods in success rate.
  • Out-of-domain Evaluation: 21.66% higher success rate than RVT-2 + DP in the OOD-object+scene split demonstrates the strongest reported compound-shift advantage.
  • Out-of-domain Evaluation: The proposed method achieves the highest success rate across all OOD splits and generally lower stability and smoothness metrics than RVT-2 + DP.
  • Qualitative Results: Qualitative rollouts show minimal sloshing and consistent upright pose during in-domain simulation transport.

C. Real-world Evaluation

The framework is evaluated on a real robot across varied cups and trials without real-world adaptation, providing a zero-shot test of synthetic-data transfer.

  • Deployment: A latency-aware asynchronous strategy overlaps inference with action execution and temporally ensembles remaining future actions.
  • Evaluation Protocol: All policies use only synthetic demonstrations, while evaluation covers glass, plastic, and metal cups across 30 real-world trials.
  • Sim-to-real Transfer: The results support zero-shot sim-to-real transfer of stability-aware behavior learned from synthetic demonstrations.
  • Real-world Results: The method achieves the best overall real-world performance among all compared methods in Table II.
  • Real-world Results: Compared with RVT-2 + DP, the method improves task success, stability, and smoothness during real-world execution.

D. Ablation Study

The ablation study shows that physically validated filtering, hierarchical keyframe prediction, sufficient proprioceptive history, and temporal ensembling each contribute to robust performance under compound distribution shifts.

  • Stability-aware data filtering: Removing trajectory filtering degrades both success rate and stability, showing that geometric and kinematic feasibility alone cannot ensure stable dynamics.The full filter combines tilt screening and fluid simulation for trajectory-level dynamic validation.
  • Hierarchical keyframe prediction: Removing the high-level keyframe policy causes a substantial performance drop, indicating that low-level control alone struggles with task goals and smooth executable trajectories.The hierarchy separates semantic reasoning from motion generation under compound shifts.
  • Proprioceptive history length: Shortening proprioceptive history from Hist-20 reduces success rates, while Hist-12 preserves comparable stability, highlighting the value of temporal context under distribution shifts.Hist-20 achieves the strongest overall performance among the compared history lengths.
  • Temporal ensemble: Removing temporal ensembling causes the most significant performance degradation among the evaluated component ablations.

VI. CONCLUSION

The paper presents a synthetic-to-real framework for smooth, dynamically stable transport of liquid-filled containers. It improves success, trajectory stability, and motion smoothness in simulation and the real world, including zero-shot sim-to-real transfer without real-world demonstrations.

  • VI. CONCLUSION: The framework combines stability-aware synthetic data generation with a hierarchical latent diffusion policy for dynamically stable and smooth liquid-container manipulation.
  • VI. CONCLUSION: Evaluations show improvements in success rates, trajectory stability, and motion smoothness over representative baselines in both simulation and the real world.
  • VI. CONCLUSION: Real-world experiments demonstrate zero-shot sim-to-real generalization without any real-world demonstrations.
  • VI. CONCLUSION: Future work will extend the framework beyond liquid transport to other dynamically sensitive manipulation tasks.
Loading 2609.18119v1…