Source-linked AI summary
Learning to Place New Objects in a Scene
Yun Jiang, Marcus Lim, Changxi Zheng, Ashutosh Saxena
TL;DR
The paper studies how robots can place diverse, previously unseen objects into semantically appropriate areas while maintaining stability and handling stacking. It learns placement preferences and geometric relationships from point-clouds through a graphical model with ILP inference. The approach achieves 98% stable placement for known objects and 82% for new objects in robotic experiments, while supporting multi-object scene organization.
Problem
Robots must choose stable, semantically preferred locations and orientations for varied objects and placing areas, including cases not seen during training.
Method
The method learns features and a graphical model from noisy point-clouds to encode stability, semantic preferences, stacking, object-area relationships, and feasibility constraints.
Results
98% stable placement was achieved for known objects and 82% for new objects in robotic experiments.
Takeaways & Limitations
The approach supports placing multiple objects across diverse areas and improves end-to-end performance over the best baseline.
Takeaways & Limitations
The method captures only limited contextual information and omits material attributes such as friction and fragility; haptic and force sensing remain future improvements.
Abstract
from arXiv · showhide
Placing is a necessary skill for a personal robot to have in order to perform tasks such as arranging objects in a disorganized room. The object placements should not only be stable but also be in their semantically preferred placing areas and orientations. This is challenging because an environment can have a large variety of objects and placing areas that may not have been seen by the robot before. In this paper, we propose a learning approach for placing multiple objects in different placing areas in a scene. Given point-clouds of the objects and the scene, we design appropriate features and use a graphical model to encode various properties, such as the stacking of objects, stability, object-area relationship and common placing constraints. The inference in our model is an integer linear program, which we solve efficiently via an LP relaxation. We extensively evaluate our approach on 98 objects from 16 categories being placed into 40 areas. Our robotic experiments show a success rate of 98% in placing known objects and 82% in placing new objects stably. We use our method on our robots for performing tasks such as loading several dish-racks, a bookshelf and a fridge with multiple items.
I. INTRODUCTION
The paper addresses robotic placing in complex scenes, where objects must be stable, semantically appropriate, and sometimes stacked despite noisy, incomplete perception. It introduces a learning-based graphical-model approach and evaluates it across varied objects, areas, and robotic tasks.
- Motivation: A valid placement must jointly satisfy stability, semantic preference, and stacking requirements while respecting dynamically changing environments.Previously placed objects can alter the structure available to later placements.
- Approach: A graphical model encodes geometric relationships, human placement preferences, stability, stacking, and physical feasibility constraints.Its parameters are learned with max-margin learning, and inference is formulated as an integer linear program solved using an LP relaxation.
- Evaluation: The evaluation covers 98 household objects from 16 categories and 40 placing areas, including single-object tests and complete scenes with up to 50 objects.End-to-end experiments report improvements over the best baseline on stability, semantic correctness, and human overall impressions.
- Motivation: Robotic placing remains less studied than grasping, especially for complex scenes containing diverse areas such as dish-racks, tables, and closet rods.Prior placing work largely focused on flat horizontal surfaces.
- Approach: The method learns from noisy, incomplete point-clouds and can place objects or areas not previously seen by the robot.The system uses depth-camera point-clouds rather than assuming complete geometric information.
III. ROBOT PLATFORMS AND SYSTEM
The system combines depth-based point-cloud perception, learned placement inference, and robot path planning. It scores sampled placements for stability and orientation, then executes the highest-scoring feasible candidate.
- Perception: The robots perceive objects and scenes as point-clouds using Kinect depth cameras, although the measurements may be noisy and incomplete.The Kinect depth image has 640x480 resolution and an operating range of 0.8m to 3.5m.
- Robot platforms: PANDA is a 6-DOF Adept arm with a parallel-plate gripper, while POLAR uses a 7-DOF Barrett arm with a three-fingered hand and mobile base.Both platforms use Kinect sensors for scene perception.
- Robot platforms: PANDA lacks force and tactile feedback, so small positioning errors can cause placement failures.Its average Kinect sensor-arm calibration error is 3mm, and estimated gripper repeatability is 0.1mm.
- Realization: The overall pipeline consists of perception, inference, and realization, with path planning moving and releasing the object at the predicted destination.Multiple-object tasks additionally require moving between objects and areas and avoiding obstacles.
- Inference: For each task, the system samples candidate placements, extracts features, scores them with the learned model, and selects the highest-scoring candidate.A placement is represented by a 3D location and 3D rotation.
B. Multiple-Object Placements
The placing problem selects semantically suitable areas, poses, and stacking relationships for multiple objects represented by point-clouds. The approach uses features encoding stability, preferred orientation, geometry, support, and gravity caging.
- Problem: Multiple-object placement selects each object's pose, placing area or stacked support, and relative 3D location from point-cloud inputs.The output is a final scene layout specified by each object's configuration, selected support, and relative location.
- Feature goals: The learned features target stable support and semantically preferred orientations, including examples such as vertical plates and upside-down glasses.Stability features are designed to remain invariant to translation and rotation about gravity when relative geometry is unchanged.
- Stability features: Supporting-contact features describe contact spread, falling distance, covariance structure, and the relationship between the object's center and the contact hull.The feature set also includes the contact-hull to object-hull area ratio.
- Geometric features: The method also models relative location and object-area geometry using point-cloud histograms and height features.Single-object experiments use 145 features: 12 for supporting contacts, 37 for caging, and 96 for histograms.
- Caging features: Gravity-caging features capture placements that confine an object within an area even when it is not strictly immovable.The method partitions the environment around the object into zones and computes height and distance features across those zones.
B. Max-Margin Learning
The paper formulates placement prediction as supervised classification over labeled placement features. It uses a soft-margin SVM to learn a separating function for good and bad placements.
- Formulation: The learning problem uses labeled good and bad placements represented by feature vectors to determine whether a placement is good.The labels are yi ∈ {−1, 1}, with 1 denoting a good placement.
- SVM model: The method builds its learning algorithm on support vector machines with theoretical performance guarantees for classification.The SVM formulation is a soft-margin problem with slack variables.
- Optimization: The learned SVM finds a separating hyperplane that maximizes the margin between positive and negative placement examples.The same max-margin learning principle is used in the graphical model for multiple-object placements.
C. Shared-sparsity Max-margin Learning
Different object-area pairs require both shared and setting-specific information, so the paper uses shared-sparsity max-margin learning. Task-specific models retain self-owned features while sharing parameters across related tasks.
- Motivation: Different placing settings have distinct properties, making neither one universal model nor fully independent models sufficient.A universal model can underrepresent setting differences, while independent models can overfit.
- Task structure: With M objects and N areas, the method treats the MN object-area pairs as tasks whose models share some underlying parameters.This structure is intended to capture both common and setting-specific placement information.
- Shared sparsity: Each task weight vector is decomposed into self-owned features and shared features, with self-owned features constrained to remain sparse.The decomposition is θi = Si + Bi.
- Inference: At test time, different task models vote to determine the best placement.The approach therefore combines task-specific predictions when ranking candidates for a new scenario.
- Scope: The approach improves performance with new objects in new areas but requires one model per object-area pair, limiting scalability.The paper identifies this model-per-pair requirement as a limitation for many objects and placing areas.
A. Data
The experiments evaluate placement learning across object-area scenarios, feature combinations, heuristics, and learning algorithms. Results show that learning methods outperform heuristics, shared sparsity helps in the hardest new-object/new-area setting, and combined features perform best.
- Data: The single-object dataset contains 8 objects, 7 placing areas, 1800 random placements per object-environment split, and 37655 collision-free placements overall.Placements were labeled using rigid-body simulation before supervised learning.
- Scenarios: The evaluation covers SESO, NENO, SENO, and NESO scenarios to test combinations of seen and unseen objects and environments.NENO excludes both the test object and placing environment from training.
- Metrics: The study evaluates ranking, classification, stability, and preference using R0, P@5, AUC, Pstability, and Ppreference.Pstability and Ppreference are robotic-arm success rates for stable and preferred placements, respectively.
- Feature evaluation: Combining all three feature types gives the best results under every evaluation metric in the SESO feature study.Each feature combination outperforms chance, but the combined representation performs best.
- Method comparison: All learning methods outperform heuristic rules under all evaluation metrics, while heuristic performance varies substantially across non-trivial placing areas.The chance baseline has Prec@5=0.1 and AUC=0.49.
- NENO results: 2.1 is the average R0 for shared sparsity SVM in NENO, compared with 8.9 for joint SVM and 5.4 for independent SVM with voting.The result shows the shared-sparsity model is strongest among the listed NENO methods, though difficult object-area cases remain.
F. Robotic Experiments
The robotic experiments evaluate learned placement under real sensing and scene constraints, including stability, preferred orientation, stacking, and multiple placing areas. Performance is higher with known object models and degrades with incomplete point-clouds and unseen objects.
- Unseen objects: 82% stable and 72% preferred-placement performance was achieved in NENO experiments without object models.Preferred placement additionally required semantically correct orientations.
- Failure analysis: Incomplete point-cloud capture caused failures when objects struck placing-area geometry such as dish-rack spikes.The authors analyzed this sensing effect separately by registering raw clouds to parameterized object models.
- Known object models: 98% success was achieved in SESO placing with a known library of object models.The robot failed only once, when a martini glass missed a narrow stemware holder after grasping displacement.
- Multiple-object placing: The approach supports multiple-object scenes with direct placement or chain stacking under full-coverage, non-overlap, and acyclic constraints.Chain stacking permits series stacking but limits each object to one supporting or supported object.
A. Graphical Model
The graphical model represents a placing strategy through object-object stacking, object-environment placement, locations, and orientations. Its potential decomposes placement quality into stability and semantic preference while retaining scene-level constraints.
- Strategy representation: A placing strategy is represented by binary stacking variables S, binary environment-placement variables T, orientations C, and locations L.The model assigns each object a base relationship, 3D orientation, and 3D location relative to that base.
- Objective: The model maximizes a potential over strategies subject to constraints discussed for physically feasible placement.Higher potential corresponds to better placing strategies.
- Graphical structure: The Markov network assembles single-placement structures sharing object, environment, location, and orientation nodes.This factorization is intended to make learning and inference tractable.
- Placement factors: Each placement includes latent binary variables for stability and semantic preference.These variables distinguish geometric stability from whether the object-base relationship matches semantic preferences.
- Placement factors: Stability depends on local geometric pose and location, whereas semantic preference depends on the object and base independently of placement details.The base configuration can also affect the context for placing another object.
B. Features
The feature design combines geometric and visual descriptions of objects and bases, with semantic features constructed to remain invariant across placements within the same placing area. These features support both stability and object-area preference modeling.
- Semantic features: Semantic features depend only on the object and base so they remain invariant to different placements within the same placing area.This separates semantic object-area compatibility from placement-specific geometry.
- Feature design: The representation includes 3D Zernike descriptors, FPFH bag-of-words, color histograms, curvature histograms, and overall-shape ratios.These capture shape, local geometric surfaces, color, curvature, and covariance structure.
- Feature design: Color features contribute 46 dimensions through hue-saturation and intensity histograms.The hue-saturation histogram is 6×6 and the intensity histogram has 10 bins.
- Feature dimensions: The full feature construction reaches 801 features, including pairwise products and minima for object and base descriptors.The summarized semantic feature dimensions are reported in Table IV.
- Experimental settings: Multiple-object experiments use relative-height caging with n_r=1 and n_θ=4 and an 81-bin grid without histogram ratios.These are experiment-specific feature settings.
C. Max-margin Learning
The method learns stability and semantic-preference weights with max-margin supervision, then performs constrained inference over discretized placement candidates. Inference is formulated as an ILP and solved in practice through LP relaxation.
- Learning: Max-margin learning estimates separate weight vectors for stability and semantic-preference feature functions.Training placements provide ground-truth stability and semantic-preference labels.
- Learning objective: The objective separates the potential of a good strategy from every alternative and introduces slack variables for training errors.The margin is intended to increase confidence in the learned model.
- Inference: Inference selects the highest-potential placing strategy while enforcing coverage, capacity, non-overlap, and acyclicity constraints.The constraints ensure objects are assigned places and conflicting placements are not jointly selected.
- Inference: The constrained inference problem is formulated as an ILP and solved using LP relaxation, which works well in practice.Special cases reduce to dynamic programming or maximum matching, while the general ILP is NP-complete.
- Inference: The large search space is discretized by sampling locations and orientations before optimization.Experiments sample locations at about 10 cm × 10 cm intervals and generate 24 orientations through 45-degree rotations.
VIII. EXPERIMENTS ON PLACING MULTIPLE OBJECTS
The experiments take raw object and scene point-clouds as input and produce a placing strategy specifying each object's location and orientation, which can then guide robotic execution.
- Method: Raw point-clouds of the objects and scene are transformed into a placing strategy containing each object's location and orientation.The inferred strategy can be used to construct the post-placement scene and guide the robot through path planning.
- Evaluation: The evaluation covers single-object, semantic-area, multiple-object, complete-scene, and robotic placing settings.Robotic tasks included loading a bookshelf and a fridge.
A. Data
The experiments evaluate learning-based placement from real and incomplete point-clouds across diverse objects, areas, stability settings, and semantic preferences.
- Data: 98 objects from 16 categories and 40 placing areas form the main evaluation database.Objects are observed with a Kinect from five views, while environments use a single-view point-cloud and no assumed 3D object model.
- Data: Thin racks and holders may use synthetic point-clouds, while the object inputs remain real and incomplete because of reflections, self-occlusions, and top-only observation.A footnote specifies that synthetic point-clouds are generated from tri-meshes for some thin racks and holders.
- Stability evaluation: The raw-point-cloud stability test uses four challenging scene types and 620 manually labeled placements.Because each scene has one placing environment, only stability features are evaluated.
- Baselines: The baselines randomly sample valid placements, vertically oriented placements, or horizontally oriented placements.The vertical heuristic targets cases such as plates in dish-racks, while the horizontal heuristic targets relatively flat areas such as tables.
- Stability evaluation: Learning algorithms outperform all baseline heuristics on AUC, although the vertical heuristic performs perfectly for pen-holder placements.The horizontal heuristic performs poorly because few evaluated placing areas are flat.
- Stability evaluation: Combining all stability features with a polynomial kernel achieves an average AUC of 0.96.The combined feature set performs best across all reported metrics in this experiment.
- Semantic preference: For semantic-area selection, the study tests 98 objects from 16 categories against 11 placing areas.Performance is weaker for clothes and shoes, whose greater variation makes learning harder; polynomial SVM with all features reaches an average AUC of 0.90 versus a 0.72 baseline.
- Multiple-object placement: Multiple-object tests evaluate 14 plates in a dish-rack, 17 objects on a small table, and five clothing items on a rod.The tasks specifically assess stacking and the LP relaxation; the resulting layouts align plates, form semantic piles, and arrange clothes on the rod.
E. Placing in Various Scenes
The method is evaluated in multi-area office and apartment scenes and on robots performing household placement tasks, with improved end-to-end performance but limited contextual modeling.
- Scene evaluation: The multi-scene evaluation uses point-clouds from three offices and two apartments, with humans judging stability and semantic correctness.Tables, cabinets, floors, and drawers are segmented as placing areas.
- Qualitative results: Figure 10 shows mostly correct placement across a dish-rack, table, and hanging rod, with errors involving two top plates and one hanger.The examples illustrate vertical plates, stacked books and plates, and aligned clothes.
- Scene evaluation: The learning algorithm substantially improves end-to-end performance under all metrics compared with vertical, horizontal, and configuration-prior heuristics.Baselines choose placing areas randomly because they use no semantic cues.
- Scene evaluation: The generated layouts adapt placements to available space, including moving some objects from a small table to the ground.Books are stacked on a table in one scene and placed horizontally on a couch in another.
- Limitations: The method does not model object-object co-occurrence preferences, such as placing a mouse beside a keyboard or grouping objects by category.The authors identify contextual relations as a future way to improve placements.
- Robotic experiments: In five trials placing 16 plates per trial, fewer than 5% of 80 placements failed because plates moved within the gripper after grasping.This experiment used a dish-rack and placed plates one by one after inferring a strategy.
- Robotic experiments: Robotic experiments include dish-rack loading, table and dish-rack placement, bookshelf and fridge loading, and recycling-bin placement.The experiments use both PANDA and POLAR robots; Figure 12 shows bookshelf and fridge arrangements.
- Overall results: The conclusion reports 83% stable-placement accuracy, 82% semantic-placement accuracy, and robotic success rates of 82% for new objects stably and 72% with semantically preferred orientations.These results summarize end-to-end and robotic performance across the evaluated settings.