Source-linked AI summary
Deformable Object Manipulation under Partial Observability via Real-Time Full-Shape Estimation
Kosar Behnia, Ville Kyrki, Gokhan Alcan
TL;DR
Deformable-object manipulation is difficult because high-dimensional, underactuated states are only partially observable, yet controllers need accurate full-shape predictions. The paper proposes cRVAE, a conditional recurrent variational autoencoder that estimates full state from sparse corner measurements for receding-horizon control. In simulation, it matches a parameter-identified XPBD reference while running far faster, and it completes real-hardware rope runs without observed obstacle contact.
Problem
Deformable objects have high-dimensional, underactuated dynamics and partial observability, making accurate full-state prediction difficult for collision-free control.
Method
cRVAE is a conditional recurrent variational autoencoder that estimates full deformable-object state from sparse corner-node measurements and provides a differentiable forward model for receding-horizon control.
Results
At horizon h = 40, cRVAE reached 2.51 cm on rope and 0.98 cm on fabric, versus 3.28 and 1.44 cm for parameter-identified XPBD, while completing 10/10 real-hardware rope runs without observed obstacle contact.
Takeaways & Limitations
Full-state estimation from sparse corner sensing at in-loop speed supports obstacle-aware deformable-object manipulation on hardware.
Takeaways & Limitations
Future work must extend cRVAE to model contact and add vision-based interior measurements to improve predictions of unmeasured nodes.
Abstract
from arXiv · showhide
Manipulating deformable objects (DOs) is challenging due to their high-dimensional state space, underactuated dynamics, and partial observability. In this paper, we propose cRVAE, a lightweight conditional recurrent variational autoencoder that estimates the full DO state from only partial corner-node observations during inference. The resulting model is used as the forward model in a receding-horizon optimal control framework for obstacle-aware collaborative DO manipulation. In simulation on rope and fabric, cRVAE estimates the full DO state from the available corner-node measurements alone, matching the accuracy of a parameter-identified XPBD model. At inference it uses no physical parameters as model inputs and performs no online parameter identification. It also runs approximately 350 times faster on the rope and over 1500 times faster on the fabric per forward pass, keeping horizon-based planning within the 100 ms control budget where XPBD exceeds it already at short horizons. Full-shape estimation from corner sensing at in-loop speed is what makes the model deployable on hardware, which we demonstrate on a Unitree Go2 robot.
I. INTRODUCTION
Deformable-object manipulation requires models that predict full shape despite high-dimensional, underactuated, partially observed dynamics. The paper introduces cRVAE, which infers full state from sparse corner measurements and supports fast, differentiable receding-horizon control.
- I. INTRODUCTION: Full-shape estimation lets the controller anticipate collisions from corner measurements and lift the robot arm to avoid an obstacle during collaborative rope manipulation.Without the full-shape estimate, the rope hits the obstacle.
- I. INTRODUCTION: Sparse sensing creates a practical need for models that infer full shape, correct estimates with measurements, and remain lightweight during repeated receding-horizon optimization.Dense observations can be expensive and difficult to maintain on physical systems, while physics-based predictions can drift under model mismatch and sim-to-real error.
- I. INTRODUCTION: cRVAE estimates full deformable-object state from sparse corner-node observations and is integrated into receding-horizon optimal control for collaborative, obstacle-aware manipulation.The model is designed for partial observability while retaining differentiability and lightweight computation for repeated planning.
- I. INTRODUCTION: During inference, cRVAE uses a learned conditional prior and injects measurement feedback into both the estimated state and recurrent hidden state.This design reduces estimation error when measurements arrive at a high rate.
- I. INTRODUCTION: ≈9 ms per rope rollout versus 3.2 s for XPBD (≈360×), enabling in-loop horizon-based control within the required real-time budget.Both timings include controller derivatives; the resulting controller completes narrow-gap tasks where a reactive baseline fails.
II. RELATED WORKS
Prior work seeks deformable-object predictors that are full-state, differentiable, fast, and usable with sparse sensing. Existing learning and physics-based approaches each address parts of this need but may require dense observations, incur computational cost, or suffer from model mismatch.
- Full-State Prediction Models for DOs: Learning-based models compress high-dimensional deformable-object observations into latent dynamics representations using architectures including MLPs, linear models, RBFs, recurrent state-space models, and graph-based models.
- Predictive Models for Optimization-Based DO Control: Differentiable control models must be evaluated repeatedly, but existing approaches may require full node observations or incur expensive gradient computation through physics simulators.
- Partial Observation and Feedback in Real-World DO Manipulation: Physics-based predictors can be interpretable and computationally efficient, yet limited feedback allows model mismatch and sim-to-real errors to accumulate over time.
- Problem Formulation: The deformable object is represented as N nodes, while inference observes only K < N boundary or corner nodes rather than the full state.
- Control-oriented deformable-object prediction must estimate full future states from sparse observations while remaining lightweight for repeated receding-horizon optimization.The formulation includes an internal state summarizing past observations and predictions.
IV. PROPOSED METHOD
The proposed cRVAE is a conditional recurrent variational autoencoder designed to infer full deformable-object states from sparse measurements while supporting differentiable receding-horizon control. It trains with full simulated states but switches to a conditional prior and measurement feedback during inference.
- cRVAE instantiates a predictor that infers full deformable-object states from sparse measurements, retains temporal information, and remains differentiable and lightweight for receding-horizon optimization.
- During inference, cRVAE replaces the full-state encoder with a learned conditional prior and decodes the full state from hidden state, sparse measurements, and manipulation input.
- Measurement feedback corrects both the estimated state and recurrent hidden state to limit drift during multi-step prediction.
a) Encoder:
The cRVAE encoder uses a hidden-state-conditioned latent prior, while the decoder reconstructs the full next state from hidden state and partial observations. Training combines reconstruction accuracy with KL-based consistency between posterior and prior.
- a) Encoder:: The prior estimates the latent distribution conditioned only on the hidden state.
- a) Encoder:: The decoder reconstructs the full next state using the hidden state and partial observations.
- a) Encoder:: The hidden state is updated at each step to support recurrent state estimation.
- a) Encoder:: Training minimizes a combination of reconstruction loss and KL divergence.
- a) Encoder:: The weighting parameter λ balances reconstruction accuracy against consistency between the encoder posterior and conditional prior, while w_c weights the corner-node term.
B. Inference with Measurement Feedback
cRVAE uses residual feedback between measured and estimated corner nodes to correct both the full-state estimate and recurrent hidden state. The corrected estimate initializes each receding-horizon planning cycle, while the recurrence remains anchored through the corrected hidden state.
- The residual between measured and estimated corner nodes addresses inference drift caused by model errors and the sim-to-real gap.
- Measurements are obtained every Tm steps and propagated between updates using a known function, such as robot kinematics or dynamics.
- Residual feedback corrects both the estimated full state and hidden state, using Jacobians and gains to regulate trust between model predictions and measurements.
- The measurement-corrected current-state estimate initializes the receding-horizon problem at each planning cycle, while the recurrence is anchored through the corrected hidden state.
C. Control Problem Formulation
The learned model Φθ is embedded in a receding-horizon optimal-control problem that optimizes manipulation inputs while enforcing obstacle constraints. The formulation supports repeated planning with a forward model and corner-node propagation.
- C. Control Problem Formulation: At each planning cycle, the controller solves a receding-horizon optimal-control problem over H steps using the learned model Φθ.
- C. Control Problem Formulation: The optimized input sequence minimizes running and terminal costs that can encode tracking toward a desired deformable-object configuration.
- C. Control Problem Formulation: Obstacle-aware manipulation is enforced through inequality constraints defining the feasible region O ⊂R3.
- C. Control Problem Formulation: The dynamics constraint uses Φθ as the forward model, while corner-node inputs are propagated within the planning horizon.
- V. EXPERIMENTS: The framework evaluates cRVAE across rope simulation, fabric simulation, and real-world rope manipulation on a Unitree Go2 robot.
- V. EXPERIMENTS: The same cRVAE architecture is used throughout, with a 491k-parameter network combining MLP encoder, prior, decoder, and GRU components.
A. Rope Simulation
In rope simulation, cRVAE predicts full-shape dynamics from corner observations and remains more accurate than geometric and XPBD references across the tested horizon. Ablations show benefits from the latent prior and measurement feedback, especially at short measurement intervals.
- a) cRVAE Training and Modeling Accuracy:: The rope experiment trains on XPBD-generated dynamics with 33 nodes, two corner measurements, and trajectories spanning circular, translational, and random motions.
- a) cRVAE Training and Modeling Accuracy:: 0.81 cm at h = 1 and 2.51 cm at h = 40, keeping cRVAE below XPBD and the geometric baseline across the horizon.At h = 1, XPBD and the baseline reach 2.75 and 2.79 cm; at h = 40, they reach 3.28 and 5.66 cm.
- b) Ablation of the Latent Variable:: 0.57 cm at h = 1 and 0.55 cm at h = 10, cRVAE’s advantage over the deterministic variant on a separate held-out set.The variants use matched protocols and parameter counts within 4%, so these values are comparable only to each other, not to Fig. 3.
- c) What Do the Corner Measurements Anchor?:: 13.1% ± 7.2 reduction in interior MAE at Tm=1 and 10.4% ± 8.7 at Tm=5 with full feedback, improving all nine held-out trajectories.The corresponding MAE changes are 1.256 to 1.058 cm and 1.226 to 1.058 cm, respectively, with p = 0.004.
- c) What Do the Corner Measurements Anchor?:: At Tm=10, no feedback condition is statistically distinguishable from the no-feedback baseline, with p ≥0.16.
- c) What Do the Corner Measurements Anchor?:: 0.35 cm interior MAE when fitting the full state versus 2.04 cm when fitting only the two corners, showing limited interior identification from corner measurements alone.Adding node 12 reaches 0.56 cm, while the damped correction remains near an already-plausible prediction.
d) Comparison:
In the baseline scenario, both methods avoid collisions at every tested speed, while in the challenging narrow-gap scenario cRVAE succeeds across speeds where the reactive comparator fails.
- Baseline Scenario: 100% of runs are completed by both methods at every speed in the baseline scenario.Both methods maintain positive clearance from the obstacles.
- Challenging Scenario: cRVAE completes the challenging task across all speed scenarios with approximately 0.10 m clearance, while fails and reaches zero clearance.The task requires planning rope deformation ahead to pass through the narrow gap.
- Challenging Scenario: At t = 5.2 s, cRVAE stretches the rope along the y-axis to clear the gap, whereas collides with the lower obstacle.Before and after the obstacle, the two rope shapes are near each other.
- Compute Cost: The rope compute-time comparison varies per horizon h, with Fig. 5 reporting log-scale prediction-plus-derivative costs for cRVAE and XPBD.The total includes forward prediction and derivative computation for each model.
B. Fabric Simulation
In fabric simulation, cRVAE predicts full-cloth states from sparse corner observations with accuracy comparable to XPBD while remaining fast enough for horizon-based control.
- cRVAE Training and Modeling Accuracy: At h = 40, cRVAE reaches 0.98 cm MAE versus 1.44 cm for XPBD and 4.92 cm for the baseline.At h = 1, the corresponding errors are 0.66, 0.57, and 4.02 cm.
- cRVAE Training and Modeling Accuracy: XPBD is roughly 1550× slower per forward pass than cRVAE, exceeds the 100 ms budget at h = 3, and cRVAE remains at 1.5 ms at h = 40 including derivatives.The compute gap grows with the number of cloth nodes and constraints.
- Comparison: The fabric task uses nodes 14 and 224 as leaders, nodes 0 and 210 as followers, and four obstacle scenarios with repeated MPPI trials.The leaders maintain a desired offset while navigating vertically constrained passages.
- Comparison: cRVAE completes all fabric runs by stretching the fabric through narrow gaps, while CBF-QP success drops to 0% for gaps g ≤1.6 m.For wide gaps, cRVAE has larger obstacle clearance but higher tracking error than CBF-QP.
a) Experimental Setup:
The real-world setup uses a Unitree Go2 with two manipulators and sparse rope sensing, followed by simulator calibration and decoder-only cRVAE calibration.
- Experimental Setup: The Unitree Go2 carries a D1 manipulator that actuates one rope end, while an SO-101 manipulator holds the other end as leader.The rope is 180 cm long and six motion-capture sensors measure positions along it.
- Experimental Setup: During inference, cRVAE uses the two rope endpoints as corner nodes despite six motion-capture markers being available for state capture.Nodes 0 and 5 provide the corner-node inputs.
- Experimental Setup: The evaluation reports decoder-calibration MAE, signed z-error per marker, and computation time per control action across real-world scenarios.Successful runs are counted out of 10 per scenario and values are reported as mean ± standard deviation.
- cRVAE Training and Calibration: The XPBD simulator calibrates stiffness, damping, mass, rest length, and sensor positions against one real-world dataset using differential evolution.This calibration reduces the sim-to-real gap before cRVAE training.
- cRVAE Training and Calibration: cRVAE is trained on calibrated 31-node rope simulations, then only its decoder is retrained on real data while encoder, prior, and recurrent weights remain frozen.Prior sampling bypasses the encoder because real data provides six markers rather than the full 31-node state.
- cRVAE Training and Calibration: Stretching and bending constraints supply gradient signals for unmeasured rope nodes during calibration, alongside marker and endpoint losses.The predicted full state contains 93 values, while measured marker positions contain 18 values.
d) Real-World Control Experiments:
The real-world experiments evaluated cRVAE in two obstacle geometries using collaborative rope manipulation, with measurements arriving at every control step. Across 10 runs per scenario, the robot completed the task without observed rope–obstacle contact, while future work targets contact modeling and richer interior measurements.
- Experimental setup: The controller planned over horizon 5 while receiving new corner measurements at every control step.The Unitree Go2 served as follower and the SO-101 arm as leader along a circular arc.
- Real-world results: 10/10 runs across the two scenarios showed no observed rope–obstacle contact during real-world manipulation.Each scenario used 10 runs with different initial robot and obstacle positions.
- Simulation comparison: In simulation at horizon h = 40, cRVAE reached 2.51 cm on rope and 0.98 cm on fabric, compared with 3.28 and 1.44 cm for parameter-identified XPBD.A geometric baseline reached 5.66 cm on rope and 4.92 cm on fabric.
- Future directions: Future work will extend cRVAE to model object–environment contact and incorporate vision-based interior measurements.Interior measurements are intended to improve prediction accuracy for unmeasured nodes.