Source-linked AI summary
Where2Act: From Pixels to Actions for Articulated 3D Objects
Kaichun Mo, Leonidas Guibas, Mustafa Mukadam, Abhinav Gupta, Shubham Tulsiani
TL;DR
Where2Act addresses the problem of inferring actionable information for interacting with articulated 3D objects, beyond passive semantic perception. It learns per-pixel actionability, interaction proposals, and success likelihoods from interaction data with adaptive sampling, and shows localized representations that generalize to novel shapes and unseen categories. The framework is evaluated in SAPIEN across six primitives and 972 shapes from 15 indoor categories.
Problem
Visual perception systems need to infer where and how to interact with articulated objects, because semantic labels alone do not specify executable actions.
Method
The framework predicts per-pixel actionability, interaction proposals, and success likelihoods from visual input, learning from interaction outcomes with online adaptive sampling.
Results
The learned actionable representations are highly localized and generalize to novel shapes and unseen object categories in SAPIEN evaluations.
Takeaways & Limitations
Where2Act connects visual perception to low-level manipulation by mapping pixels on articulated parts to candidate actions and their expected outcomes.
Takeaways & Limitations
The final online data procedure samples half its trajectories randomly because prediction-biased sampling may insufficiently explore novel regions.
Abstract
from arXiv · showhide
One of the fundamental goals of visual perception is to allow agents to meaningfully interact with their environment. In this paper, we take a step towards that long-term goal -- we extract highly localized actionable information related to elementary actions such as pushing or pulling for articulated objects with movable parts. For example, given a drawer, our network predicts that applying a pulling force on the handle opens the drawer. We propose, discuss, and evaluate novel network architectures that given image and depth data, predict the set of actions possible at each pixel, and the regions over articulated parts that are likely to move under the force. We propose a learning-from-interaction framework with an online data sampling strategy that allows us to train the network in simulation (SAPIEN) and generalizes across categories. Check the website for code and data release: https://cs.stanford.edu/~kaichun/where2act/
1. Introduction
Where2Act targets actionable visual understanding for articulated objects by predicting where and how short-term manipulations should occur and whether they will succeed. It learns these predictions from interaction data and evaluates generalization across shapes and categories.
- Motivation: The task is to infer possible interactions, their locations, and the motions needed to perform them on novel objects.Examples include pulling a drawer handle, pushing a door, flipping a switch, and pressing a button.
- Motivation: Passive semantic representations such as labels and part structure provide limited understanding of executable actions.The paper motivates action-oriented perception as a complement to semantic visual representations.
- Approach: The proposed network predicts per-pixel actionability scores, interaction proposals, and success likelihoods for an atomic action type.Atomic interactions include primitives such as pushing and pulling, decomposed into where and how to act.
- Approach: The framework learns from interaction outcomes and uses on-policy data sampling to obtain more informative training examples than random interaction alone.Successful actions are labeled by whether they cause a desirable state change.
- Evaluation: 972 shapes across 15 indoor object categories and six primitive interactions are used in SAPIEN to train and test the approach.The reported evaluations include novel objects and previously unseen categories.
- Contributions: The contributions include a per-pixel articulated-object affordance task, interaction-based adaptive sampling, and SAPIEN benchmarking environments.The paper reports generalization to novel shapes and unseen object categories.
2. Related Works
Prior work infers semantic or geometric properties and learns affordances from passive observation or interaction, but often does not model the low-level actions required for richer articulated-object interactions. Where2Act addresses this gap by predicting both affordances and executable interaction details.
- Semantic representations: Semantic labels, keypoints, and part segmentations can identify relevant object structure but do not specify what action or precise movement should follow.The paper gives handles and buttons as examples of structures whose actionable use still requires additional knowledge.
- Geometric and physical properties: Geometric and physical representations can support classical planners by describing shape, pose, articulated parts, or functional semantics.These representations are presented as inputs to off-the-shelf planning techniques.
- Passive observations: Passive-observation methods learn affordances from demonstrations, but anatomy mismatch and imitation-learning distribution shift make the approach challenging.Passive data may lack sufficiently varied positive examples and negative examples.
- Learning by interaction: Interaction-based learning has succeeded for simpler tasks such as planar grasping, segmentation, pushing, and non-planar grasps.These systems are described as limited in the complexity of the actions they model.
- Learning by interaction: Where2Act predicts affordances for richer articulated-object interactions while also learning the low-level actions that induce the desired change.The approach is enabled by scalable simulation with diverse objects.
3. Problem Statement
Where2Act formulates actionable information as dense per-pixel predictions for articulated 3D objects. For each primitive, the system predicts where to interact, how to interact, and the likelihood of a successful outcome.
- Task formulation: The task predicts per-pixel actionable information for manipulating articulated 3D objects under different action primitives.The outputs cover interaction location, interaction method, and outcome.
- Inputs and actions: Inputs can be a 2D RGB image or a 3D partial point cloud scan of the articulated shape.The framework considers six short-term primitive actions parameterized by robot gripper pose in SE(3).
- Success definition: An interaction is successful when valid contact with the intended point causes considerable part motion along the intended direction.The evaluation uses directional agreement and thresholds on articulated-part motion magnitude.
- Predictions: For each visible articulated-part pixel or point, the model predicts an actionability score measuring how likely it is to be actionable.This is the location component of the actionable-information prediction.
- Predictions: For each pixel or point, the model proposes interaction parameters and assigns a success likelihood to each proposal.The proposals specify candidate ways to interact with the selected point.
4. Method
Where2Act maps RGB images or partial 3D scans to localized predictions of where and how to interact, and whether proposed actions will succeed. Its interaction-trained system combines three decoding modules with adaptive sampling to improve learning from sparse successful trials.
- Network Modules: An encoder-decoder backbone extracts dense per-pixel features from either RGB images or partial 3D point clouds.The 2D implementation uses UNet, while the 3D implementation uses PointNet++.
- Network Modules: Three decoding heads predict pixel actionability, multiple gripper-orientation proposals, and success confidence for each proposal.Actionability scores lie in [0,1], proposals are generated from random Gaussian noise, and proposal scores also lie in [0,1].
- Network Modules: The action proposal module is a conditional generative model that maps pixel features and sampled noise to diverse 3-DoF gripper orientations in SO(3).The orientation uses a 6D rotation representation formed from the first two orthonormal axes of a rotation matrix.
- Collecting Training Data: Training data come from SAPIEN interactions in which a flying gripper executes six pre-programmed SE(3) primitives and records whether each position-orientation tuple succeeds.Offline random sampling bootstraps the data, while online sampling adds trials over regions favored by the current network predictions.
- Collecting Training Data: Only 1% of randomly sampled pulling interactions are positive, motivating adaptive sampling that concentrates trials on likely successful subregions while retaining 50% random exploration.The final procedure mixes random and prediction-biased trajectories equally to reduce sampling inefficiency without eliminating exploration.
- Training and Losses: The training objective combines action scoring, action proposal, and actionability losses, with the proposal loss using a Min-of-N strategy to match one candidate to each positive orientation.Separate positive and negative data queues are used to address interaction-outcome imbalance.
5. Experiments
Experiments use SAPIEN to evaluate six primitive interactions across articulated objects, comparing baselines, ablations, and learned predictions on novel categories and real-world data. Results show that 3D-Ours performs best, online sampling improves performance, and predictions capture action- and orientation-specific geometric cues.
- Framework and Settings: The SAPIEN environment evaluates six primitive actions across 972 shapes from 15 indoor object categories, including novel-category testing.The experiments use 10 categories for training and reserve five for testing.
- Framework and Settings: A successful interaction requires considerable part motion in the intended direction, with directional agreement validated when the angle is below 60°.The intended direction depends on the primitive, such as forward or backward for pushing and pulling, and up or left for directional variants.
- Metrics and Baselines: The sample-success rate measures the fraction of network-proposed interaction trials that succeed, jointly evaluating actionability, proposal, and action-scoring modules.The evaluation samples pixels, generates 100 proposals per pixel, samples an orientation, and checks the simulated outcome.
- Metrics and Baselines: 3D-Ours performs best against B-Random, B-Normal, and B-PCPNet, while 2D-Ours remains competitive despite lacking explicit 3D information.The comparisons include testing shapes from both training categories and unseen test categories.
- Metrics and Baselines: Online adaptive sampling improves performance over the ablated model without online sampling in most cases.The strategy samples more interaction data from regions predicted to be likely successful.
- Results and Analysis: Predictions are action-specific and gripper-aware: pulling emphasizes handles and high-curvature boundaries, whereas pushing highlights pushable flat surfaces and suppresses collision-prone handle regions.Directional primitives also learn distinct interaction orientations, such as pushing-left on drawer sides and pulling-up on bucket handles.
- Results and Analysis: Networks trained on synthetic data produce promising action-scoring predictions on real 3D scans and 2D images.This indicates transfer of learned actionable information to real-world data within the reported evaluation.
6. Conclusion
The paper formulates per-pixel actionable prediction for articulated 3D objects and trains networks to map object pixels to primitive actions and interaction parameters. Results indicate localized learned representations generalize to unseen categories, while several scope limitations remain.
- The task predicts per-pixel actionability and candidate interaction parameters for six primitive actions on articulated object parts.
- The networks learn highly localized actionable representations that generalize to novel unseen object categories.
- Limitations and Future Works: Single-frame visual input can leave articulated-part mobility ambiguous when motion information cannot be determined from one snapshot.
- Limitations and Future Works: The experiments cover six action primitives with hard-coded motion trajectories rather than free-form interactions.
- Limitations and Future Works: The method does not explicitly model part segmentation and part motion.
A. Framework and Settings: More Details
The framework uses standardized SAPIEN simulation settings, articulated-object assumptions, predefined robot trajectories, and validity checks to collect interaction outcomes. Trials are successful only when the intended interaction is valid and produces sufficiently directed part motion.
- Environment: The simulation uses fixed-root articulated objects whose movable parts are convex-hull approximated and settled before interaction.Objects are initialized with random articulated-part poses, then simulated until final rest states.
- Action Settings: Six pre-programmed action trajectories drive interactions through a velocity-controlled robot gripper with kinematic interpolation.The gripper may start either closed or open, with motion controlled through NVIDIA PhysX's internal PID controller.
- Interaction Validity: Interaction trials are invalid unless the direction lies in the positive surface-normal hemisphere and the gripper initially avoids object collision.Pushing additionally requires first contact between the closed gripper and target articulated part, with fingers rather than the hand making contact first.
C. Training Details and Computational Timing
Training uses Adam with a 0.001 learning rate and modality-specific inputs, while data collection and convergence require several days. Inference is substantially faster, taking 4 ms for a batch of 32 RGB/depth images.
- Training Details: Training uses a 0.001 learning rate with Adam, without image-based augmentation, while randomly down-sampling 3D point clouds.Images use 224 × 224 inputs and point clouds use 10000 × 3 inputs.
- Computational Timing: Offline interaction collection takes 3–4 days and training takes 4–5 days until convergence, whereas inference takes 4 ms for 32 RGB/depth images.
- Dataset Coverage: The SAPIEN evaluation covers 15 object categories, with one visualized example per category in Figure 8.
E. More Results on Real-world Data
Figure 9 presents additional results from applying the networks directly to real-world data.
- Figure 9 visualizes additional real-world results obtained by directly applying the networks.
F. Actionability Scoring Predictions: More Result Visualization
Figure 10 visualizes additional actionability-scoring predictions for the six action primitives.
- Additional examples show the actionability-scoring module’s predictions across six action primitives.
G. Action Proposal Predictions: More Result Visualization
Figure 11 visualizes additional action-proposal predictions for example shapes under each action primitive.
- Additional examples show action-proposal predictions for example shapes across each action primitive.
H. Failure Cases: Discussion and Visualization
The paper illustrates failure cases that expose task difficulty and ambiguity, alongside visualizations of simulation assets, real-world results, and action predictions.
- Failure Cases: Failure cases include invalid initial gripper-object collisions, insufficient part motion, and difficult interaction configurations.The caption describes these cases as demonstrating task difficulty and ambiguity.
- Failure Cases: Some cases are intrinsically ambiguous for a robot to resolve from a single static snapshot.
- Simulation Assets: Simulation assets include one visualization example for each of 15 object categories.
- Real-world Results: Real-world results use 3D scans and web images, with predictions shown over all pixels because articulated-part masks are unavailable.Predictions outside articulated parts are not guaranteed, although the results make sense when motion is allowed for entire objects.
- Prediction Visualizations: Additional visualizations cover actionability scores and top-10 action proposals for the six action primitives.Motion trajectories in the action-proposal visualization are exaggerated 3×, and some examples contain no predicted actionable proposal.