PaperlayerView exact revision on arXiv

Source-linked AI summary

VAT-Mart: Learning Visual Action Trajectory Proposals for Manipulating 3D ARTiculated Objects

Ruihai Wu, Yan Zhao, Kaichun Mo, Zizheng Guo, Yian Wang, Tianhao Wu, Qingnan Fan, Xuelin Chen, Leonidas Guibas, Hao Dong

arXiv:2106.14440v2cs.CVcs.RO

TL;DR

Articulated objects are challenging to manipulate, and standard kinematic representations can omit geometry and functionality relevant to interaction. VAT-Mart learns object-centric affordances and trajectory proposals through interaction, and the authors report generalization to unseen shapes, categories, and real-world data.

  • Problem

    Standard visual articulation representations such as joint parameters and part poses can abstract away geometric and functional cues relevant to manipulating articulated objects.

    Source: §1 · p. 2

  • Method

    VAT-Mart jointly trains an RL policy to explore manipulation trajectories and a perception module to learn dense, task- and interaction-aware affordances and trajectory proposals.

    Source: §1 · p. 2

  • Results

    Experiments on 562 PartNet-Mobility shapes across seven categories report generalization to unseen shapes, novel object categories, and real-world data.

    Source: §1 · pp. 2–3

  • Takeaways & Limitations

    The learned priors provide object-centric guidance about where and how to interact, including trajectory proposals conditioned on the task and interaction type.

    Source: §1 · p. 2

    The current open-loop trajectory prediction uses a single-frame input, and the work considers only one-degree-of-freedom articulation.

    Source: §3 · p. 4, §6 · p. 9

Abstract

from arXiv · show

Perceiving and manipulating 3D articulated objects (e.g., cabinets, doors) in human environments is an important yet challenging task for future home-assistant robots. The space of 3D articulated objects is exceptionally rich in their myriad semantic categories, diverse shape geometry, and complicated part functionality. Previous works mostly abstract kinematic structure with estimated joint parameters and part poses as the visual representations for manipulating 3D articulated objects. In this paper, we propose object-centric actionable visual priors as a novel perception-interaction handshaking point that the perception system outputs more actionable guidance than kinematic structure estimation, by predicting dense geometry-aware, interaction-aware, and task-aware visual action affordance and trajectory proposals. We design an interaction-for-perception framework VAT-Mart to learn such actionable visual representations by simultaneously training a curiosity-driven reinforcement learning policy exploring diverse interaction trajectories and a perception module summarizing and generalizing the explored knowledge for pointwise predictions among diverse shapes. Experiments prove the effectiveness of the proposed approach using the large-scale PartNet-Mobility dataset in SAPIEN environment and show promising generalization capabilities to novel test shapes, unseen object categories, and real-world data. Project page: https://hyperplane-lab.github.io/vat-mart

1 INTRODUCTION

Articulated objects are difficult to perceive and manipulate because their many degrees of freedom complicate interaction, while standard visual articulation models can omit geometry and functionality relevant to manipulation. VAT-MART learns object-centric affordances and trajectory proposals through interaction, with experiments reporting generalization to unseen shapes, categories, and real-world data.

Motivation

  • Articulated objects’ higher-DoF state spaces make them harder to understand and interact with than rigid objects.The introduction notes that articulated parts such as cabinet drawers and doors are frequently interacted with by people and agents.Source: §1 · p. 1
  • Standard kinematic structures, part poses, and joint parameters can abstract away interaction hotspots and part functionality needed for manipulation.Examples include edges, holes, and bars, as well as handles and doors.Source: §1 · p. 2

Contributions

  • VAT-MART predicts dense per-point affordances and trajectory proposals adapted to geometry, interaction type, and task.The object-centric proposals abstract away robot configurations and scene context, aiming to serve different manipulation setups.Source: §1 · p. 2, Figure 1 · p. 2
  • The framework jointly trains an RL policy to discover successful interaction trajectories and perception networks to summarize and generalize those discoveries.Perception feedback also encourages the RL policy to explore diverse trajectories.Source: §1 · p. 2
  • Experiments use 562 PartNet-Mobility shapes from seven categories in SAPIEN and report generalization to unseen shapes, novel categories, and real-world data.The authors attribute the reported generalization to large-scale training over diverse, textureless geometry.Source: §1 · pp. 2–3

2 RELATED WORK

Prior work estimates standardized articulation representations for manipulation, while actionable visual representations instead predict information tied to downstream actions. VAT-Mart extends dense affordance maps with task-aware trajectory proposals and learns them from interaction data.

Perceiving and Manipulating 3D Articulated Objects

  • Earlier perception and manipulation systems commonly use link poses, joint parameters, and kinematic structures, which may miss fine-grained geometry such as drawer handles and faucet switches.Prior work also explored learning articulation states and structures, and using learned visual knowledge for manipulation.Source: §2 · p. 3

Learning Actionable Visual Representations

  • Actionable visual representations predict action probabilities for manipulation, unlike visual semantics defined independently of a task.Examples include grasping and manipulation affordances, as well as spatial maps, keypoints, and contact points.Source: §2 · p. 3

Learning Actionable Visual Representations; Learning Perception from Interaction

  • Following Where2Act, VAT-Mart uses dense affordance maps and adds task-aware trajectory distributions beyond task-less short-term manipulation.Interaction trajectories discovered by an RL policy supervise the perception system, which returns curiosity feedback to encourage diverse exploration.Source: §2 · pp. 3–4

3 ACTIONABLE VISUAL PRIORS: Action Affordance and Trajectory Proposals

The paper’s actionable visual priors jointly indicate where and how to interact with an articulated object, conditioned on interaction type and task. They predict per-point actionability, trajectory distributions, and trajectory success likelihoods.

Actionable visual priors

  • Each articulated-object point receives an actionability score, a distribution of visual action trajectories, and a success-likelihood score for each trajectory.The actionability and success scores lie in [0,1], and the trajectory distribution can produce diverse proposals.Source: §3 · p. 4
  • Predictions are conditioned on the object, interaction type, and task, so they indicate whether an interaction can succeed and propose how to perform it.The paper gives pushing or pulling and opening a door by a target angle as examples.Source: §3 · p. 4

Inputs

  • The input object is a partial point cloud, and this work considers pushing and pulling on parts with one degree of freedom.Revolute-joint tasks use angular targets in radians; prismatic-joint tasks use length targets relative to global shape scale.Source: §3 · p. 4

Outputs

  • A trajectory is a sequence of up to five 6-DoF end-effector waypoints, represented using the first waypoint and subsequent waypoint-to-waypoint changes.Each waypoint contains a 3-DoF hand center and a 3-DoF orientation.Source: §3 · p. 4

4 VAT-MART: AN Interaction-for-perception LEARNING FRAMEWORK

VAT-MART learns actionable visual priors through an RL policy that explores manipulation trajectories and a perception system trained on the resulting interactions. Curiosity feedback connects the two components by encouraging exploration of trajectories the perception model scores poorly.

4 VAT-MART: AN Interaction-for-perception LEARNING FRAMEWORK

  • The framework couples an RL policy that explores trajectories with a perception system that learns actionable visual priors from the collected interactions.The policy supplies training data to perception, while perception provides curiosity feedback to encourage diverse solutions.Source: §4 · p. 4, Figure 2 · p. 5

4.1 THE RL POLICY FOR INTERACTIVE TRAJECTORY EXPLORATION

  • For each interaction and articulation type, a conditional TD3 policy collects trajectories across shapes, contact points, and task specifications.In simulation, RL training has access to ground-truth state information; the RL network is discarded at test time, when perception predicts the priors.Source: §4.1 · pp. 4–5
  • The RL policy receives a task-completion reward, stepwise reward for moving the part toward its target, and a penalty for moving away from the contact point.Trials stop at task success or after five waypoint steps.Source: §4.1 · p. 5

4.2 THE PERCEPTION NETWORKS FOR ACTIONABLE VISUAL PRIORS

  • The perception network predicts per-point actionability, a distribution of trajectory proposals, and a success likelihood for each trajectory.Its inputs include a partial point cloud, contact point, task, and—when scoring a trajectory—the trajectory itself.Source: §4.2 · p. 6
  • The actionability map scores where to interact, while a conditional variational autoencoder generates diverse trajectories from Gaussian noise.A separate scoring module estimates the success likelihood of a proposed trajectory.Source: §4.2 · p. 6

4.3 CURIOSITY-DRIVEN EXPLORATION

  • Perception returns curiosity feedback to RL by rewarding trajectories that receive low predicted success scores, encouraging exploration of novel interactions.New interaction data then continues to supervise perception, and proposals from different training epochs are retained to avoid mode chasing.Source: §4.3 · pp. 6–7
  • Table 1 evaluates learned visual priors on test shapes from training categories and test categories, reporting each metric with the two results separated by a slash.The caption states that higher values indicate better results but supplies no metric values here.Source: Table 1 · p. 7
  • Figure 3 presents qualitative examples as an input shape and task, an actionability heatmap, and three trajectory proposals at a selected contact point.These examples show the intended distinction between predicting where to interact and proposing how to interact.Source: Figure 3 · p. 7

5 EXPERIMENTS

Experiments evaluate actionable visual priors and downstream manipulation in simulation, then examine generalization to real-world data. The method outperforms baselines on most reported manipulation comparisons and shows some generalization to real-world depth scans.

5 EXPERIMENTS

  • Evaluation uses SAPIEN and PartNet-Mobility, with 562 shapes across seven categories and tests on held-out shapes and categories.The experiments cover doors and drawers and split applicable categories into training and test categories.Source: §5 · p. 7

5.1 ACTIONABLE VISUAL PRIORS

  • Trajectory scoring is evaluated with accuracy, precision, recall, and F-score; proposal coverage measures whether collected trajectories have a sufficiently similar prediction.The scoring test set contains 350 positive and 350 negative trajectories per experiment.Source: §5.1 · pp. 7–8
  • Actionability heatmaps identify interaction locations, while trajectory proposals provide diverse ways to accomplish desired tasks.Examples include pulling a drawer from its handle or pushing from inside, and using varied trajectories to push a door closed.Source: §5.1 · p. 8

5.2 DOWNSTREAM MANIPULATION

  • For articulated-object manipulation, Table 2 compares our method with three baseline methods using task success rates within 15% tolerance to tasks, and our method outperforms baselines on most comparisons.Source: §5.2 · p. 8, Table 2 · p. 9
  • Naive RL largely fails; the heuristic wins only on drawer pushing, while Where2Act consistently performs worse.The heuristic uses ground-truth handle masks and joint parameters; Where2Act assumes an oracle part-pose tracker and uses segmented interactions.Source: §5.2 · pp. 8–9

5.3 REAL-WORLD EXPERIMENTS

  • A model trained on synthetic textureless data generalizes to real-world depth scans to some degree; the paper also shows a real-robot experiment.Examples include a microwave, a cabinet, and a captured 3D cabinet scan.Source: §5.3 · p. 9, Figure 5 · p. 9

6 CONCLUSION

The paper concludes with object-centric actionable visual priors for articulated-object manipulation, combining dense affordance predictions with visual trajectory proposals. It identifies single-frame open-loop prediction as a limitation and describes the RL policy architecture.

6 CONCLUSION

  • Object-centric actionable visual priors combine dense action affordance predictions with diverse visual trajectory proposals for manipulating 3D articulated objects.The paper reports experiments on PartNet-Mobility and real-world data as evidence of the approach’s effectiveness.Source: §6 · p. 9
  • Current open-loop trajectory prediction uses a single-frame input; future work may consider multiple frames, improve performance, and study more interaction and articulation types.The authors describe this work as a first attempt at learning these representations.Source: §6 · p. 9

A.1 NETWORK ARCHITECTURE DETAILS

  • The RL policy uses a four-layer MLP for both its policy and Q-value networks, predicting a residual gripper pose and a single Q value, respectively.The policy network receives a 33-dimensional state and outputs a 6-dimensional action; the Q-value network receives 39-dimensional state-action input.Source: §A.1 · p. 16

A.2 MORE IMPLEMENTATION AND TRAINING DETAILS

The implementation specifies training hyperparameters and a perception-network architecture that maps geometry, contact, trajectory, and task features to actionability, trajectory proposals, and success scores.

A.2 MORE IMPLEMENTATION AND TRAINING DETAILS

  • For RL policy training, the buffer size is 2048, batch size is 512, and Adam learning rate is 0.0001 for both the policy and Q-value networks.Source: §A.2 · p. 16

B.1 NETWORK ARCHITECTURE DETAILS

  • Four encoders process point-cloud, contact-point, trajectory, and task features; three heads predict actionability, trajectory proposals, and trajectory success likelihood.The trajectory proposal head uses a conditional variational auto-encoder to generate diverse trajectories from sampled Gaussian values and conditioning features.Source: §B.1 · pp. 16–17

B.2 MORE IMPLEMENTATION AND TRAINING DETAILS

The perception networks learn from RL interaction data, while curiosity-driven exploration seeks trajectories the scorer is less confident about. The appendix also describes training targets and dataset visualizations.

B.2 MORE IMPLEMENTATION AND TRAINING DETAILS

  • The scorer is trained on 5,000 successful and 5,000 unsuccessful RL trajectories; the proposal module learns from successful trajectories, and actionability targets use the mean of the top five scores among 100 sampled trajectories.The scorer uses binary cross-entropy, the proposal module uses rotation and translation losses plus KL regularization, and actionability uses L1 loss.Source: §B.2 · p. 17
  • Perception-network training uses batch size 32 and an Adam optimizer learning rate of 0.001 for all three modules, as the note’s stated hyperparameters.Source: §B.2 · p. 17

C MORE DETAILS FOR THE CURIOSITY-DRIVEN EXPLORATION (SEC. 4.3)

  • Curiosity rewards penalize trajectories the scoring module already rates as likely to succeed, encouraging exploration of less-confident trajectories.Perception training balances successful and unsuccessful trajectories and samples successful trajectories equally from high- and low-likelihood groups.Source: C MORE DETAILS FOR THE… · p. 17
  • Table D.3 summarizes shape counts in the dataset, while Figure D.6 shows one example shape from each of the seven object categories used in the paper.Source: Table D.3 · p. 18, Figure D.6 · p. 18

D MORE DETAILED DATA STATISTICS

The appendix documents dataset and simulation settings, computational costs, evaluation protocols, and baseline implementations, including the limitations of hand-crafted heuristics.

D MORE DETAILED DATA STATISTICS; E SAPIEN SIMULATION ENVIRONMENT SETTINGS

F COMPUTATIONAL COSTS AND TIMING

  • Around 3 days per experiment were required on one Nvidia V100; inference takes 15 ms for actionability and 8 ms each for trajectory proposals and success scoring.The reported training stages took around 18, 15, 10, and 20 hours, respectively.Source: F COMPUTATIONAL COSTS AND TIMING · pp. 18–19

G.1 EVALUATING ACTIONABLE VISUAL PRIORS (SEC. 5.1)

  • Evaluation uses precision, recall, F-score, and average accuracy; trajectory coverage uses a weighted position-orientation distance with threshold 10 and averages results over 10 runs.The distance is D(l1,l2) = 5Dpos(l1,l2) + Dori(l1,l2).Source: §G.1 · p. 19

G.2 EVALUATING DOWNSTREAM MANIPULATION (SEC. 5.2)

  • For downstream tasks, the protocol generates 100 trajectories at the highest-actionability contact point per shape and counts success within a 15% task-specification tolerance.This yields 350 trajectories for each door-pushing, door-pulling, drawer-pushing, and drawer-pulling task.Source: §G.2 · p. 19

H.1 NAIVE RL BASELINE

  • The naive RL baseline uses one module to predict the initial gripper waypoint and a second to predict subsequent waypoints from initial and current scene states.Both modules receive the task specification; the second also receives the initial and current gripper poses.Source: §H.1 · pp. 19–20

H.2 RULE-BASED HEURISTIC BASELINE

  • Rule-based heuristics require task-specific human engineering and can fail with absent or occluded handles, intricate grasps, and dynamic effects such as inertia.Drawer pushing samples the front board, while drawer pulling targets the handle and counts missing handles as failures.Source: §H.2 · p. 20, Figure H.7 · p. 21

H.3 WHERE2ACT BASELINE

  • Where2Act samples among top-five actionable points, selects its top-rated action, then executes short 0.05-unit motions for up to 15 steps using an oracle part-pose tracker.For pulling, it closes the gripper after moving forward and moves backward only after a successful grasp.Source: §H.3 · pp. 20–21

I ABLATION STUDY ON CURIOSITY FEEDBACK

The ablation compares the final approach with a version lacking curiosity feedback and reports that curiosity feedback improves prediction performance in most entries.

I ABLATION STUDY ON CURIOSITY FEEDBACK

  • Curiosity feedback improves prediction performance in most reported entries compared with the version that removes it.The table reports results separately for test shapes from training categories and shapes from test categories; higher values indicate better results.Source: I ABLATION STUDY ON CURIOSITY FEEDBACK · p. 21

J MORE RESULTS AND ANALYSIS

Additional qualitative analyses show actionability and trajectory proposals, trajectory scoring predictions, and different trajectories arising from different contact points for the same task.

J MORE RESULTS AND ANALYSIS

  • Figure J.8 presents additional qualitative results for the actionability prediction and trajectory proposal modules, augmenting Figure 3 in the main paper.Source: J MORE RESULTS AND ANALYSIS · p. 22
  • Figure J.9 presents an additional qualitative analysis of the learned trajectory scoring module, serving as a supplement to Figure 4 in the main paper.Source: J MORE RESULTS AND ANALYSIS · p. 22
  • For the same task, interacting at different contact points over a shape produces different trajectory proposals.Figure J.10 presents the input shape and task, predicted actionability heatmap, and proposals at three contact points.Source: J MORE RESULTS AND ANALYSIS · p. 22

K REAL-ROBOT SETTINGS AND EXPERIMENTS

The real-robot experiments use a Franka Panda facing a cabinet, with a front-right RGB-D camera providing point clouds to the learned models.

K REAL-ROBOT SETTINGS AND EXPERIMENTS

Loading 2106.14440v2…